Table of Contents

Team LiB
Next Section
Oracle9iR2 Data Warehousing
Oracle9iR2 Data Warehousing
by Lilian Hobbs, Susan Hillson and Shilpa Lawande ISBN:1555582877
Digital Press © 2003 (519 pages)

This book will show you how to use the Oracle database with tools such as Oracle Discoverer to query the warehouse, generate reports that can be deployed over the web, and gain better insight into your business.

Table of Contents Back Cover
Table of Contents
 | Oracle 9iR2 Data Warehousing
 | Foreword
 | Preface
 | Chapter 1 | - | Data Warehousing
 | Chapter 2 | - | Designing a Warehouse
 | Chapter 3 | - | Data Warehouse Features
 | Chapter 4 | - | Summary Management
 | Chapter 5 | - | Loading Data into the Warehouse
 | Chapter 6 | - | Oracle Warehousing Tools
 | Chapter 7 | - | Managing the Warehouse
 | Chapter 8 | - | Data Warehousing and the Web
 | Chapter 9 | - | Oracle 9i OLAP and Data Mining
 | Appendix
 | Index
 | List of Figures
 | List of Tables
Team LiB
Next Section


Back Cover

Oracle9iR2 Data Warehousing
Oracle9iR2 Data Warehousing
by Lilian Hobbs, Susan Hillson and Shilpa Lawande ISBN:1555582877
Digital Press © 2003 (519 pages)

This book will show you how to use the Oracle database with tools such as Oracle Discoverer to query the warehouse, generate reports that can be deployed over the web, and gain better insight into your business.

Table of Contents Back Cover
Back Cover

Oracle9IR2 Data Warehousing is a guide to using the Data Warehouse features in the latest versions of Oracle—Oracle9i Release 1 and Release 2. Written by people on the Oracle development team that designed and implemented the code, this updated and expanded edition provides an insider’s view of how the Oracle9i Release 2 software is best used for your application.

It provides a detailed look at a wide range of topics including:

  • new techniques to facilitate the ETL process to transform data and load the warehouse
  • How to use the Oracle OLAP and Data Mining options
  • An entire chapter devoted to using materialized views to radically improve warehouse performance.

This book will show you how to use the Oracle database with tools such as Oracle Discoverer to query the warehouse, generate reports that can be deployed over the web, and gain better insight into your business.

This how-to guide provides step-by-step instructions including screen captures to make it easier to design, build, and optimize performance of the data warehouse or data mart.

  • Based on 9i plus the improvements in Release 2
  • Presents concepts with working examples
  • Describes best practices and critical implementation tradeoffs, based on users’ actual implementations

About the Authors

Lilian Hobbs has been involved in the design and tuning of computer systems for over 20 years and holds a Ph.D. from the University of Southampton. To date she has written ten books including guides to using the Oracle and Rdb databases. Lilian is a project leader in Oracle’s Summary Management Development team.

Susan Hillson is a Senior Director of Engineering Development at Oracle. Her group is responsible for the development of Summary Management, Oracle Data Guard, and the Database Utilities. She holds a MS degree in Computer Information Systems from Boston University. Susan has over 20 years’ experience in the computer industry, including over 6 years teaching programming and database management systems. Susan co-authored Oracle8i Data Warehousing (Digital Press, 2000).

Shilpa Lawande is a principal developer in the Server Technologies Division. She is a member of the team that introduced materialized view technology into the Oracle server. Shilpa has a BA from Indian Institute of Technology, Bombay, and a Master’s degree in Computer Science from University of Wisconsin-Madison.

Team LiB


Oracle 9iR2 Data Warehousing

Lilian Hobbs
Susan Hillson
Shilpa Lawande
Digital Press An imprint of Elsevier Science
Amsterdam * Boston * London * New York * Oxford * Paris * San Diego San Francisco * Singapore * Sydney * Tokyo

Digital Press is an imprint of Elsevier Science.

Copyright © 2003, Elsevier Science (USA).

All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher.

Recognizing the importance of preserving what has been written, Elsevier Science prints its books on acid-free paper whenever possible.

Library of Congress Cataloging-in-Publication Data

Hobbs, Lilian, 1959-
  Oracle 9iR2 data warehousing / Lilian Hobbs, Susan Hillson, Shilpa Lawande.
p. cm.

ISBN 1-55558-287-7
1. Oracle (Computer file) 2. Data warehousing I. Hillson, Susan, II. Lawande, Shilpa. III. Title.

QA76.9.D37 H62 2003

658.4'038'0285574--dc21 2002041271

British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library.

The publisher offers special discounts on bulk orders of this book.

For information, please contact:

Manager of Special Sales
Elsevier Science
200 Wheeler Road
Burlington, MA 01803
Tel: 781-313-4700
Fax: 781-313-4882

For information on all Digital Press publications available, contact our World Wide Web home page at: http://www.digitalpress.com or http://www.bh.com/digitalpress

10 9 8 7 6 5 4 3 2 1

Printed in the United States of America

For Simon, sadly missed

-Lilian

For Stacey and Emily, my golden girls

-Susan

To my parents and in-laws

-Shilpa

Acknowledgments

When writing a book, there are many people who are involved at various stages of the production cycle. The extremely important but often silent contributors are those people who give up their valuable spare time to review the book. We are extremely grateful to the following people, who gave up their valuable time to review this book.

  • Srinivas Bala is Senior Manager, Quality Assurance, at Narad Networks. Srinivas has several years of experience with the design of Oracle databases and applications in the telecommunications industry.

  • Jay Davison, Director of Software Development, Database Utilities, Oracle Corporation.

  • Jean-Pierre Dijcks, Product Manager for Oracle Warehouse Builder, Oracle Corporation.

  • Mike Durran, Discoverer Product Manager, Oracle Corporation.

  • Bud Endress is Director of Product Management for Oracle OLAP Products and has contributed screenshots and examples for the OLAP chapter.

  • Abhinav Gupta is Principal Member of Technical Staff in the Server Technologies division at Oracle.

  • Ramkumar Krishnan is Principal Member of Technical Staff in the Oracle Data Mining Group at Oracle.

  • James Lear, a member of the Discoverer development team at Oracle.

  • David Martland is a Technical Architect with the Britannia Building Society in the United Kingdom, who has many years experience using the Oracle database.

  • Judy Muller, Assistant Director, Business Services-University of New Hampshire.

  • Jack Raitto, Consulting Engineer and Project Leader for Summary Management at Oracle.

  • Pete Smith, is an independent Oracle data warehousing consultant. Based in the United Kingdom, he has many years experience, gained from working on various Oracle database projects.

  • Jim Steiner, Senior Director of Product Management, Oracle Corporation.

The previous edition of this book was also reviewed by Jay Feenan, Lory Molesky, Simon Oxbury, Michael Schmitz, Dennis Murray, and Mike Rubino.

We also need to say a big thank you to:

  • Steve Hagan, Vice President of the New England Development Center, for his encouragement and support throughout the writing of this book.

  • Max Hillson, whose e-commerce site, coolstuffcheap.com, served as the inspiration for many of the shopping examples.

  • Chuck Rozwat, Executive Vice President of Oracle Server Technologies, for his support while writing this book.

Finally, this book is dedicated to all the software developers at Oracle around the world, who write the software that made this book possible.

Our thanks also go to Theron Shreve and Pam Chester at Reed Elsevier and Alan Rose and his team for actually producing this book.


Foreword

Three years ago, at the dawn of the Internet revolution, I wrote the foreword to the first edition of this book. Today, the infrastructure that we saw emerging is solidly in place and is being used in ways we could not have predicted just 36 short months ago.

Intelligent management of information is essential to the challenges we face in business and in ensuring our safety and security in daily life. Data warehouses are uniquely able to bring together information to identify trends, and understand how to maximize effective use of resources.

The ability to capture accurate, usable, and timely information, integrate it with other data, analyze it, and deliver it via the web to employees, customers, suppliers, and partners has become key to running a successful enterprise. Heretofore, data warehouses were used to aid in the strategic decision making process of a business. Today, they are also being used for tactical decision making, and to support day-to-day operations. While data warehouses were once considered "nice to have," twenty-first century organizations cannot run effectively without them.

Data warehouse growth, both in usage, and size, is accelerating rapidly.

Today, Oracle data warehouses frequently contain over a terabyte of data. Within the next five years some organizations will find themselves managing petabytes of data. The Internet has enabled instant, on demand accessibility of information, to a much larger set of user populations. Some Oracle data warehouses in operation today already have thousands of online users.

The days of waiting overnight for a report to run are long gone, along with the concordant slow decision-making. By using Oracle 9i, pre-summarizing data and storing it in materialized views, reports that used to take 2 hours to run, now complete in 2 minutes. Some Oracle sites run over 150,000 reports and queries in a single day. With new analytic capabilities in the Oracle 9i database, OLAP queries can execute directly on data stored in the Oracle data warehouse.

Oracle 9i warehouses are now loaded daily, and even multiple times a day using transportable tablespaces, SQL*Loader, external tables, and partition maintenance operations, rather than updating the warehouse periodically in batch, on weekends.

Data warehouses have become the source of data for critical applications including customer relationship management (CRM) and supply chain management (SCM). They are increasingly being used for fraud detection as well. Data Mining capabilities, which have been incorporated into the Oracle database can be used to help detect if the activity of an account increases suddenly or usage patterns change.

Whether you are designing a large data warehouse, small data marts, or looking for tips on how to improve performance using Oracle 9i Release 2, this book will provide you with advice and examples you can follow.

Recommended techniques are illustrated with numerous working examples in each chapter. Written by Lilian Hobbs, Susan Hillson, and Shilpa Lawande, members of the Oracle Server Technologies development team, this book offers a thorough coverage of the data warehousing features in Oracle 9i Release 2, the tools in Oracle 9i Application Server and Oracle 9i Developer Suite used to build, access, and manage your data warehouse.

Chuck Rozwat
Execute Vice President
Server Technologies
Oracle Corporation


Preface

There was a time when it was rare for a company to have a data warehouse, but now it is an essential source of information to ensure that businesses make the right decisions. Initially, data warehouses could only be built and maintained by large corporations, but the recent advances in database software and computer hardware mean that now any organization can build a data warehouse or a data mart.

This book introduces the reader to the subject of data warehousing and data marts, with respect to implementing them using an Oracle 9i Release 2 (9iR2) database. It has been designed so that anyone new to this subject can gather a basic understanding of data warehousing and what is involved in designing, creating, implementing, and managing this solution using Oracle 9iR2.

We have endeavored to make the book easy to read and have provided lots of examples to illustrate how features are implemented. Each chapter focuses on a specific topic, and, by reading each chapter, you will learn the flow and all the basic steps required to build a data warehouse using Oracle 9iR2. The structure of the chapters is as follows:

  • Chapter 1 provides an introduction to the subject of data warehousing—introducing terms and discussing the need for a data warehouse or data mart.

  • Chapter 2 shows how to design a data warehouse and illustrates how to create dimensions and fact tables. At that time, the EASYDW shopping example is introduced; it is used for all of the examples in this book.

  • Chapter 3 describes various techniques that you can use to improve query performance in your data warehouse. Advice is given on how to make best use of features such as indexes and partitions.

  • Chapter 4 describes how you can achieve huge improvements in query response time by using materialized views and query rewrite.

  • Chapter 5 shows how to extract data, transform data, and then load this data into the data warehouse.

  • Chapter 6 looks at some of the Oracle tools you can use to build and report on your data warehouse. Specifically described are Oracle Warehouse Builder and Oracle Discoverer.

  • Chapter 7 discusses the techniques you can use to manage your data warehouse, such as backup and restore and managing partitions.

  • Chapter 8 describes how the Web can play a key role with your data warehouse and how it can be used to benefit your business.

  • Chapter 9 discusses the Oracle OLAP and Oracle data mining options, which are now part of the 9iR2 database and which can help you gain better insight into your business using the data warehouse.


Chapter 1: Data Warehousing

1.1 Introduction

Since we published the first edition of this book, based on Oracle 8i, we've witnessed the rise and fall of the dot-coms and followed the ups and downs of the NASDAQ. We've seen URLs become part of our everyday vocabulary and are exchanging e-mail with just about everyone we know. We changed the way we conduct business and now can buy or sell most anything we want on the Web. We can find old friends, the nearest pet store, what's playing at the movies, or a good restaurant. We are always reachable via our wireless cell phones and take notes on our PDAs. We never get lost when we've got a GPS to guide us. We use self-service applications both at work and home. We can transfer money from one account to another with the click of a mouse and no longer have worry about making it to the bank during business hours or the security of being at an ATM at night.

What's behind this technology? Almost always there's a database to organize and store the data, and often there's a data warehouse used in the strategic decision-making process of the business.


What is a data warehouse?

1.2 What is a data warehouse?

The data warehouse is a database containing data from multiple operational systems that has been integrated, aggregated, and structured so that it can be used to support the analysis and decision-making process of a business.

1.2.1 Why Do You Need a Data Warehouse?

Is all the information needed to run your business available when it's needed, in the form in which it's needed, and in sufficient detail and with accuracy to base decisions upon? Or, do two users arrive at a meeting with reports that don't match? One thinks sales for March are $500 million, and another says they're $524 million. After much analysis, you determine that different data has been used to calculate the sales in each report, and you spend considerable time trying to figure out why and correcting the problem.

Does your company have multiple systems for the same function—the old inventory system and the new one you just spent millions of dollars building? Do you need to get data from both of these to combine for reporting purposes? How well is this working? Do users need to understand the differences between the two systems to query them on line? If this requires custom programming, do you still have an application backlog?

Do you have sufficient historical detail for analysis purposes? How many months of history are you able to keep on line? Did you save the right level of detail? Did you even save all of the historical data? Are you able to analyze the sales for each product for each geographic region before and after a major reorganization of the sales force reporting structure? Data warehouses are built to help address these types of problems.


A historical perspective

1.3 A historical perspective

In the 1970s the first commercial applications were built to computerize the day-to-day operations of a business. These systems were built on mainframe computers, which were very expensive. Only large businesses could afford the hardware, the programmers to program them, and the operations staff to keep them running. These systems were focused on inserting new data and reading this data sequentially using magnetic tapes.

With the invention of disk storage, the data could be accessed directly. This led to the first database management systems, which organized the data either hierarchically or in a network. These database systems were very complex. Programmers had to understand how the data was stored on the disk and navigate through the data to generate reports. Application programmers used COBOL to create custom reports. It took several days or even weeks to write the program for each new report. Reports were printed on computer paper and manually distributed to the users. There were never enough programmers, so there was always an application backlog. Once data could be accessed directly, the first on-line transaction processing (OLTP) systems were built.

In the late 1970s and early 1980s, minicomputers such as Digital's PDP-11 and VAX 11/780 brought the hardware costs down. Data was often stored in the complex CODAYSL database, which was extremely difficult to change and hard to understand and design. All that changed with the introduction of the relational database. In 1979, the Oracle database became the first commercially available relational system.

With the relational model, data is organized into tables with columns and rows. Rather than using pointers to maintain relationships among the data, a unique value, such as customer number, is stored in multiple tables to identify the row. The relational model was much easier to understand, and SQL, the language used to access the database, did not require knowledge of how the underlying data was physically stored. It became much easier to build applications, which led to widespread use of database management systems. After the initial release of relational systems, many companies began developing products used to access relational databases, including ad hoc query, reporting, and analysis tools.

With the introduction of the PC, computing moved from mainframes to client/server systems. Oracle applications were introduced in the late 1980s. Companies no longer had to build their own custom applications; they could now purchase software that provided basic functionality from vendors such as Oracle, PeopleSoft, and SAP.

As relational databases matured in the 1980s OLTP systems were built using relational systems to automate the operational aspects of the business. These included systems such as order processing, order entry, inventory, general ledger, and accounting. OLTP systems automate processes and represent the state of a system at a current point in time. In an inventory application, there are transactions to insert new items into the inventory, delete items when sold, and update the quantity on hand, which is always maintaining the balance on hand. A limited amount of history is retained. It is easy to determine how many of product "111-45-222" is on hand, for example, or on what date order number "45321" was shipped. During this time, the relational database vendors focused on improving performance for OLTP applications and competed with each other using industry standard TPC-C benchmarks.

1.3.1 The rise of the data warehouse

Once the OLTP systems were built to efficiently collect data, the challenge became how to best interpret this data. In the late 1980s and early 1990s, in an effort to take a broader view across the entire business, the first enterprise data warehouses (a term invented by Bill Inmon, the "father of data ware-housing") were built. Data was brought together from the many operational systems used to run the day-to-day business operations in order to provide a corporatewide view.

Data warehouses were built to view the business over time and spot trends. Many decisions require being able to look beyond the details of today's operations and take a broader view of the business. Typical ware-house queries involve reporting on product sales over the last two years or looking at the impact of a major snowstorm on retail sales versus Internet sales. Queries involve looking at how values have changed over time and what else also changed, and possibly discovering connections.

In order to perform this type of analysis, data in the warehouse needs to be retained for long periods of time, often five to ten years.

The data warehouse looks beyond data to find information

In a data warehouse, the primary activity is querying, or reading, the data. The only update activity occurs when new data is loaded. Decision support systems (DSS), such as Oracle Discoverer, provide interactive querying, charting, graphs, and reporting capabilities. Oracle added new types of access structures, such as bitmap indexes, to improve query performance. Materialized views were added in Oracle 8i to improve performance when aggregating and summarizing data.

Oracle Express Server became the analytical front end to the data ware-house to provide on-line analytical processing (OLAP). OLAP software is used to analyze business data in a top-down hierarchical fashion. It assumes queries will be posed iteratively, where the results of asking one question leads to asking many more questions.

It's not enough to know just the profit made this year; analysts also need to know profit over time for each product for each geographic region. This is a three-dimensional query—the dimensions are product, time, and geographical region. Or an analyst may need to compare this month's sales to the same month last year for each store versus the Internet site. He may drill down to a more detailed level in the hierarchy to get the sales for individual stores to determine which ones are most profitable and which have lost money.

The data warehouse requires a different database design

Data warehouses are designed for quick retrieval, when the access path is not known in advance. Information is often derived from other data, by rolling up data into summaries, drilling down to get more detail, or looking for patterns and trends.

In an OLTP system, entity relationship diagramming techniques (E-R) are used to design the database schema. Each entity becomes a table, attributes become columns, and relationships are represented by joining the primary-key and foreign-key columns together at run time.

A normalized design provides optimal performance for OLTP systems, which support high volumes of transactions that frequently update data. Normalization ensures the tables are correctly formed by putting related data together in one table and eliminating redundancy. By having only one copy of the data, update anomalies are avoided. By only updating the data in one place, consistency is maintained. After normalizing the data, some redundancy may have been reintroduced on columns that were not updated, to improve performance.

In order to optimize performance for a warehouse, where the primary activity is querying the data, a new data model was needed. Ralph Kimball, the primary industry spokesperson for dimensional modeling and author of The Data Warehouse Toolkit, introduced the star schema, a new way of designing the database to facilitate OLAP processing. In order to optimize performance for a warehouse, dimensional modeling techniques are used.

The dimensional approach to modeling organizes data into fact and dimension tables. It represents data in a way that is easily understood by users. Users often ask for reports of sales results on a quarterly basis broken down by store and geographical region. The sales numbers are the facts. Store, region, and quarter are the dimensions the data is analyzed by, and are used to organize the data. With dimensional modeling, denormalization and redundancy is introduced. In Chapter 2, we will see how to actually create a design for Easy Shopping Inc., the example that will be used throughout this book.

The logical design is converted to a physical representation that will best optimize performance and manageability. Tables, constraints, indexes, and partitions are defined.

Oracle has added several features to support dimensional designs. The optimizer can recognize a star schema and performs special optimizations, which avoid the more costly full Cartesian product. In addition to creating tables and columns, you can also define dimensions.

1.3.2 Data warehouses evolved as separate systems

You may already be wondering why you can't use your operational production systems for your data warehouse. You already have databases that are accessible through your corporate network, so why can't you just use those to get the information you need to run your business more efficiently? Why do you need to copy data from one system to another to build a warehouse? Since operational systems may not contain historical data, the information may not be available to analyze. Also, the schema is not designed and the data is not structured for business intelligence queries.

In addition, queries in the warehouse typically access large amounts of data, which require a great deal of memory, CPU, and I/O resources. Running decision-support queries that require a large amount of computing power to sort and summarize millions of rows will have an impact on the performance of your operational systems.

To make it even more difficult, the many different operational systems often are running on different hardware platforms, with different operating systems and database-management systems. Some applications may have been purchased and others built in house. On some, the source code may no longer even be available to make changes to, and many of the systems could be several years old.

It was once believed that distributed databases were going to allow a user to issue a query and that global query optimizers would locate the data transparently, returning the data to the user fast enough so that he or she never realized the data was located on a machine in a different geographical location. But systems of this type never materialized.

As a result, data needs to be moved from the operational systems into a separate data warehouse, where it is stored in a common format on a different machine for analysis.

The data warehouse is built from the operational systems

Building a warehouse involves extracting data from operational systems, sometimes combining the data with additional information from third parties, transforming it into a uniform format and loading it into the database.

Once data is entered in the warehouse, it almost never changes, since the data records the facts of an event or state that existed at some moment in time, such as a particular sale of a product that happened on December 23, 1998. If there were another sale of a similar product on December 24, 1998, it would generally be recorded as a separate event.

Often up to 80 percent of the work in building a data warehouse is devoted to the extraction/transformation/load (ETL) process: locating the data; writing programs to extract, filter, and cleanse the data; transforming it into a common encoding scheme; and loading it into the data warehouse.

Operational data must be extracted from the source operational systems and copied to the staging area—a temporary location where the data is cleansed, transformed, and prepared for the warehouse. Sometimes you have direct access to the source systems; however, often access is severely restricted, and you can only get files of data that have been extracted for you. The operational systems frequently must be running on a 24/7/52 basis, and performance cannot be impacted in any way.

Data from multiple systems needs to be transformed into a common format for use in the warehouse. A knowledge of the meaning of the data in the operational system is required, including the following types of examples:

  • Each operational system may refer to the same item by a different name or key. For example, a tool company in the United States might call product_id "1234" a "wrench," while a company in another country might call the same product "1234" a "spanner."

  • Each system might use a different encoding scheme. The product_id may be represented as three characters separated by dashes (xxx-xx-xxx) in one system and characters separated by spaces (xxx xx xxx) in another. The data must be transformed to a common encoding scheme in the warehouse.

  • An attribute of a table may have different names. One system might refer to a column in the customer table as gender, represented by values "0" or "1." Another system may call it sex, represented as "M" or "F."

  • Different systems may use different units of measure. The sales amount might be in dollars in the United States and the euro in Europe. The data must be transformed to a common measure in the warehouse.

In designing the transformation process, these different column names from the operational systems are mapped into the common name chosen in the warehouse and transformed into a common encoding scheme.

Once the data is transformed, it is ready to be loaded into the ware-house. Often the transformation area is on a machine separate from the warehouse; thus, the data will need to be transported or moved to the machine where the warehouse is located.

If the transformed data is stored in a flat file, it may be transported using FTP and then loaded using the SQL*Loader utility or 9i external tables. If the data has been transformed in an Oracle database, transportable tablespaces may be used to move a tablespace from one database to another.

Oracle Warehouse Builder, a tool in the Oracle Internet Developer Suite, helps automate the extraction, transformation, transport, and load aspects of the process. We'll take a further look at the extraction and transformation process in Chapter 5.

New data is generally added to the warehouse on a periodic basis. It is usually loaded in batch in the evenings or at another time when the ware-house is not being used heavily by the analysts.

In addition to the data you already own, you can purchase data from external data providers to add to your warehouse. For example, you can buy information about the weather, demographics, and socioeconomic data.

Examples of use could include the following:

  • Adding data that tracks regional weather events on a daily basis; this way, you can determine which products show an increase or decrease in sales when there is a snowstorm.

  • Adding customer demographic data; selective marketing can be performed, targeting those customers most likely to respond to a sales promotion.

  • Knowing what types of customers buy what types of products, you can anticipate demand and increase profitability. Demographic data can be used to help choose a location to place a new retail store.

  • Adding Dun and Bradstreet data containing information on companies and products.

1.3.3 The data mart

Building a warehouse can be very complex and often takes anywhere from 18 months to three years to deploy. Because a warehouse contains many subject areas and crosses multiple organizations, it can also be highly political. Many early data warehousing projects failed.

It was discovered that many of the same benefits of a warehouse could be scaled down to the department or line of business, solving a particular business problem. Data warehouses contain multiple subjects that provide a corporate view across all lines of business. Data marts are subject-specific or application-specific data warehouses and contain data for only one line of business such as sales or marketing. The major difference between a data mart and a data warehouse is the scope of the information it contains. Because the scope of a data mart is much smaller, the data is obtained from fewer sources, and the typical time to implement a data mart is shorter.

Independent data marts

Data marts may be dependent or independent, based on the source of information. The source of information for a dependent data mart is an existing data warehouse. A data mart is considered independent when no enterprise data warehouse exists, and the data is extracted directly from the operational systems.

Because independent data marts can be constructed very quickly, they became quite popular in the mid to late 1990s as each department in a company created its own data mart for its own needs. Unfortunately, after creating a few data marts, problems began to arise. Each data mart is its own "island of information." It's obviously a problem when two reports with different answers to the same question occur.

One of the reasons independent data marts can be deployed so quickly is that they postpone some of the critical decisions that later become necessary as the number of marts grows. Only the data needed by an individual department needs to be identified and understood. A complete understanding of all the corporate data is not necessary. Creating independent data marts avoids political issues related to the creation of common naming and encoding standards.

Other problems arose from the fact that the individual data marts were often built independently of one another by different autonomous teams. These teams will often select different hardware, software, and tools to use.

Each independent data mart gets its data directly from the operational system. If a company had five different data marts, each needing customer information, there would be five separate programs running to extract data from the customer table in the operational system. You probably don't have enough time in the batch window to run five extract programs today, and you certainly won't be able to run more extract programs in the future as you add more data marts.

Each does its own data cleansing and transformations, possibly each in a slightly different way. It is very easy for the data to become inconsistent. Redundant and inconsistent data leads to different answers, making it difficult to make decisions. Imagine trying to merge these different views at a later point into a common data warehouse.

Operational data stores consolidate reporting of recent information

As discussed previously, there is a major distinction between the data in the operational systems and the data in the warehouse. Operational data is about the current state of the company and is used to manage the daily operations. Data in the warehouse is informational, containing a historical record about the past.

If there is a need to provide information about the current state of the business to make tactical decisions to support day-to-day operations, an operational data store (ODS) may be built as part of the information management architecture. An ODS contains subject-oriented, integrated data that has been validated and cleansed. It is used to support operational queries and reports. One example is customer service organizations that need to access current account balances and billing information.

The ODS may be updated in near real time, so that it reflects the current state in time. The ODS may serve as the source of data for the ware-house or data marts.

Incrementally building the data warehouse with dependent data marts

To solve these problems, and still provide a timely return on investment, rather than building a warehouse representing the entire corporation, people began building the warehouse a functional area at a time using a phased approach.

Figure 1.1 shows the most common architecture used today. Data is extracted from OTLP systems and external sources, loaded into operational data stores and enterprise data warehouses, and loaded into dependent data marts.

Click To expand
Figure 1.1: Enterprise data warehouse with dependent data marts.

Building a data warehouse is just like building software. You cannot do everything in one release, and you will never be able to anticipate all of the possible uses. It is much better to develop an overall architecture, and build a framework with components that allow the warehouse to be built in phases. Limit the scope, and plan for enhancements every three to six months.

Reporting, query, and analysis tools

In 1995 Larry Ellison, the founder and CEO of Oracle, first introduced his vision of the network computer: a small, inexpensive device that makes it easy to run applications that access information via the Internet. Although the network computer never gained significant market share, the vision of Internet-centric business computing accelerated the rapid price decline of PCs, meeting the demand for cheaper, simpler desktop computing.

In 1998 Oracle released its entire applications suite enabled for use on the Web, and stopped developing applications for use in a client/server environment. Oracle8i, the Internet database, was released with many new features designed to support Internet-based activities and applications.

The ability to publish reports on the Web can make information available to virtually anyone. It gives employees, partners, and customers realtime access to critical information. No longer do you have to be in the office to view a report. Just pop into the local Internet cafe or connect to the Internet from your hotel room. Placing information on the Web (either your internal web or the World Wide Web) means that your office can truly be almost anywhere.


Oracle 9i

1.4 Oracle 9i

Beginning in late 2000 and with the recent introduction of Oracle 9i Release 2 in 2002, Oracle released the Oracle 9i line of its core technology products: the Oracle 9i Database, Oracle 9i Application Server, 9iAS, and Oracle 9i Developer Suite. Components previously available separately have now been integrated into suites.

Oracle dramatically reworked its application server, Oracle 9iAS for the Oracle 9i release, to incorporate technology for building portals, analyzing Web traffic, and creating and launching wireless applications. Oracle 9iAS Discoverer workbooks and Oracle 9iAS reports can be published as portlets. Oracle 9iAS clickstream intelligence reports the effectiveness of a Web site using Web server logs.

In addition to advances in application server technology came improvements to the application development tools. Now bundled as Oracle 9i Developer Suite, the tools speed up the creation of Internet-based BI applications. Jdeveloper wizards provide a rapid development environment for creating crosstabs, graphs, and other elements for analysis and presentation.

1.4.1 Data warehousing features in the 9i Database

Many features were added to the Oracle 9i Database to improve the performance, manageability, and scalability of the data warehouse. These features included the following:

  • Bitmap join indexes to improve performance when joining tables

  • A new partitioning method to allow the DBA to list the data values to be stored in each partition

  • Improvements for ETL processing:

    • Change data capture provides a mechanism to identify changed rows in an Oracle database

    • External tables allow you to provide transformations while loading the data into the database

    • Merge is a new SQL command that performs upserts, useful in loading dimension data. If a row already exists, it is modified. If the row is not yet stored in the database, it inserts it.

    • Table functions provide pipelined parallel execution of PL/SQL transformations

  • Materialized views can now be incrementally (fast) refreshed after individual DML statements, as well as after direct path loads

We'll take a look at these things and more throughout the book.

Business intelligence (BI) software provides knowledge workers with access to relevant data for reporting and analysis. It encompasses end-user query and reporting tools, OLAP tools, data mining tools, and executive information systems (EIS), which provide drill down, navigation, and exception reporting. In Oracle 9i, many business intelligence functions have been incorporated into the database.

1.4.2 Oracle 9i Database and express server technology converge

OLAP was first defined by Dr. E. F. Codd, the father of relational databases. He stated that relational databases were not originally intended to provide data synthesis, analysis, and consolidation—functions being defined as multidimensional analysis. For many years separate analytical databases such as Oracle Express were necessary to provide the functionality not available in relational databases.

Oracle has added many features that facilitate OLAP queries, and with Oracle 9i Release 2, it is now possible to use the Oracle server directly for OLAP. The SQL language has been extended to provide analytical functions, such as ranking, moving window aggregates, period-over-period comparisons, ratio to report, statistical functions, inverse percentiles, hypothetical rank and distributions, histograms, and first/last aggregates. Multiple levels of aggregation can be calculated using cube, rollup, and grouping sets. Most calculations are done directly within the server. These functions allow the OLAP queries to be expressed without complex self-joins and subqueries and allow the optimizer to choose a better execution plan.

1.4.3 Data mining functionality is embedded in Oracle 9i Database

Oracle 9i Data Mining, a new option to the Enterprise Edition, embeds data mining functionality into the database for making classifications, predictions, and associations.

Data mining is part of the knowledge discovery process. By using statistical techniques, vast quantities of data can be transformed into useful information. Data is like the raw material extracted from traditional mines: When turned into information, data is like a precious metal.

Data mining extracts new information from data. It allows businesses to extract previously unknown pieces of information from their warehouse and use that information to make important business decisions.

The discovery process typically starts with no predetermined idea of what the search will find. Large amounts of data are read, looking for similarities that can be grouped together to detect patterns and trends.

OLAP and DSS tools look at predefined relationships associated with the structure of the data. These are represented by constraints and dimensions. Data mining detects relationships that are associated with the content of the data and not yet defined, such as which products are most likely to be purchased together, known as market-basket analysis. When analyzing data over time, data can be used to detect unexpected patterns in behavior. The likelihood of an activity being performed sometime after another activity can be determined. Common applications for data mining include customer retention, fraud detection, and customer purchase patterns. Data can be mined looking for new market opportunities.

OLAP tools allow you to answer questions such as: Did sales of lava lamps increase in November compared with last year? Data mining tools help to identify answers to questions such as: What factors determine the sales of lava lamps?

With OLAP tools, analysts start with a question or hypothesis and query the warehouse to prove or disprove their theory. With data mining tools, the work is shifted from the analyst to the computer. Data mining tools use a variety of techniques to solve a number of different problems.

Oracle 9iAS Personalization, a real-time recommendation application, uses Oracle 9i Database's embedded data mining functionality to dynamically serve personalized recommendations to both anonymous Web visitors and registered customers, enabling 1:1 marketing for e-businesses. Oracle 9iAS Personalization answers questions such as: Which items is this person most likely to buy or like, with what likelihood? People who bought this item are likely to buy which other item? This type of personalization can be seen at Amazon.com and is becoming common at many other Web sites.


Building a data warehouse poses many challenges

1.5 Building a data warehouse poses many challenges

Although data warehousing has matured as a technology over the last several years, data warehouse developers face new challenges as the business and technology environments change. The database must be built for performance and reliability and must be able to scale with the business as it grows.

1.5.1 Managing the warehouse

As large data warehouses are growing to many terabytes in size, with increasingly higher availability requirements, it is critical to maintain good performance for large numbers of geographically distributed users. Backup and recovery procedures must be established, and both the data content and the use or activity in the warehouse must be managed.

The decision support workload is highly variable. In an OLTP system, an application is tuned to process many identical update transactions as quickly as possible. In a data warehouse, performance must be tuned to process as many variable queries as possible.

Usage patterns provide the foundation for tuning the warehouse for better performance. Who is using what data? What levels of summarization are they looking at? What data is not being used? Is the data structured in the most efficient manner; is it indexed on the correct columns? Can the summary tables be used for most queries? If many queries join data from one table with another, it might be beneficial to denormalize the data, prejoining the data. Workload information helps determine where indexes should be added, where tables should be combined, and where summaries should be created.

Eventually, it may no longer be necessary or practical to keep all the detail data on-line for immediate access. The data may be purged without keeping a copy. Or the data can be archived and moved to some low-cost medium such as tape or CD-ROM, where it can later be retrieved if necessary.

1.5.2 The role of metadata

Metadata is data that describes the other data and operations on that data. Metadata can be used for either technical or business purposes. As data flows from the operational systems into the warehouse, it is extracted, transformed, and summarized. Technical metadata is needed to describe this process and is essential for proper "drill down" to finer levels of detail. Business metadata allows end users to determine what data is available in the warehouse or data mart and how the data can be accessed. Metadata provides the integration and uniformity of data across the corporation. It is the place where the different departments describe their use of the term "product." Metadata is stored in a repository, which is typically a set of tables in an Oracle database. It can then be shared by any user or tool.

In 2000 the Object Management Group (OMG) published the "Common Warehouse Metamodel (CWM) Specification," which defines a metadata format for all data warehouse and business intelligence products. The specification was developed jointly by several companies, including Oracle and IBM Corporation. Since the publication of the specification, many of the data warehousing products have evolved to adhere to the standard.

1.5.3 Increasing data volume

One of the biggest technology issues facing enterprises today is the explosion of data volumes, which is expected to occur over the next several years. Data warehouses are "the" very large databases. The size of databases is increasing more rapidly now than ever before. There are many reasons for this growth, including the following:

  • With hardware improvements and storage costs continuing to decline each year, it is economically feasible to keep more and more detailed historical data. You may now be able to store a record of every product a customer bought in the supermarket, not just the fact that they bought five items for a total cost of $25.75.

  • Businesses are storing more and more data for longer periods of time.

  • Data is stored multiple times for different purposes. Indexes and materialized views are created to improve query performance, but these access structures require additional storage space, further increasing the size of the database.

  • Unstructured data can be integrated with traditional business intelligence applications. Storing multimedia data increases the database size. To store one hour of video requires about 1 GB of storage. To store one minute of audio requires a little less than 1 MB. Images can range from 20 KB to as much as 60 MB depending on the type and quality of the image.

  • Documents can be tagged with XML-based metadata and stored in Oracle 9i. Clickstream data is expected to add terabytes of data to the warehouse.

But remember that a data warehouse should not be viewed as a repository for archived data; this is not its purpose.

1.5.4 Higher availability

Ensuring the availability of the data warehouse is becoming more and more mission critical for many businesses. As data warehouses are becoming more operational in nature, feeding information back to the OLTP systems in businesses that operate globally, users need access to the data warehouse 24/7.

Oracle 9i Database is designed to eliminate the need for planned downtime and withstand any failure: system failure, storage failure, site failure, or human error. If a server goes down, your applications keep running. Real application clusters (RAC) make applications scalable and highly available. A single database can be run on a group of servers clustered together. As additional servers are added to the cluster, applications can scale to support increased throughput, with no modification.

Data Guard can be used to maintain a transactionally consistent copy of the data warehouse, which can be used to ensure operations continue with minimal interruption if there were a disaster, human error, or data corruption.

1.5.5 More users/better performance

The ability to publish reports on the Web makes information available to many more people. As data warehouses and business intelligence tools make more and better data available, the number of end users continues to grow. The demand for better performance is more important than ever. In addition, the types of queries are increasing in complexity.

With the average user only staying on a Web page five seconds, information must be up-to-date and available instantaneously. In most cases it must be precomputed and cached.

1.5.6 New types of applications

Data warehouses are being used to support new types of e-business initiatives, including customer relationship management (CRM) and supply chain management. CRM helps attract new customers and develop customer loyalty, important in the retention of existing customers. A data ware-house contains the information about a company's customers and is often the integration point for sales, marketing, and customer care applications.


The future of data warehousing

1.6 The future of data warehousing

Where are we going from here? While no one can predict the future, some trends seem to be underway.

1.6.1 Real-time data warehouses

The data warehouse will evolve to support real-time analysis and decision making. Rather than updating the warehouse periodically in batch, when a transaction is committed on the OLTP system, it will become available in the data warehouse, providing the capability of real-time decision making.

This allows the warehouse to be used to support tactical as well as strategic decisions. It enables a credit card company to detect and stop fraud as it happens, a transportation company to reroute its vehicles quickly after an accident has taken place, and an on-line retailer to communicate special offers based on a customer's Web surfing behavior.

1.6.2 The disappearance of the separate data warehouse

One day we may be using a single database for both OLTP and data ware-housing. Oracle is building capabilities into the database that allow a blending of operational and analytical capabilities, many of which are available in Oracle 9i. With this approach it would no longer be necessary to have separate databases for the OLTP, ODS, data warehouse, and data marts. This would eliminate the need for huge volumes of data movement-the extraction, transformation, loading, and replication across these four databases-and reduce the cost and complexity of integrating and managing multiple databases.

Already in Oracle Applications, both OLTP and decision support and reporting are being done in the same Oracle instance using RAC. With Oracle 9i, it is no longer necessary to have a separate relational, OLAP, data mining, or ETL engine. This greatly simplifies the operation and management of the data warehouse infrastructure.

Of course, there is still the need to integrate data from many sources-until they all are stored within one Oracle database.

1.6.3 Summary

We've gone from the mainframe in the 1970s to the minicomputers in the 1980s to client/server in the 1990s. Today, Internet computing is the most significant change, because it will slash costs while improving service for every business.

With the Internet it is now possible to deploy business intelligence applications to large, geographically distributed user populations both internally and externally to suppliers and customers.

In this chapter we've taken a look at what gave rise to data warehouses and data marts, some of the highlights of Oracle 9i, some of the many challenges facing warehouse developers, and what we think the future holds. Now it's time to see how we can use all of this technology to build and access our data warehouse.


Chapter 2: Designing a Warehouse

2.1 Introduction

Readers of this chapter probably fall into one of three categories. They have:

  1. Never designed a database before

  2. Designed a database for a transaction processing-type system

  3. Built a data warehouse system

In the latter case, the reader could skip this text or use it as a refresher, especially if his or her last database used Oracle. Therefore, this chapter is aimed at readers who fall into categories one or two, which may surprise the person who has previously designed a nondata warehouse database. Why? Because the skills and techniques used to create a database for a data ware-house will be different from those required for a transaction processing-style system. Consequently, while you will have a head start because some of the techniques are the same, it is very important to say to yourself: I am designing a different type of database.

So what is different about designing a database in a data warehouse? In a transaction-processing system, the designer's goal is to make the transaction complete very, very quickly, and the designer also has the benefit of knowing how the business will interrogate and use the data. Contrast that with a data warehouse, where, although queries must complete as quickly as possible, they could still take hours.

Another major problem is determining what information should be held in the warehouse and at what level of granularity it should be retained. This book will not discuss the techniques that can be used to determine what should be included in the warehouse and how to go about collecting this data; there are already many books available that discuss this topic extensively.

However, the importance of trying to determine what should be included in the data warehouse cannot be stressed enough. It is so important because it may not be until a year after the warehouse is in production that you suddenly discover that the information is either not available or held at an inappropriate level. For example, a telephone company decides not to hold every call in its database but instead holds a total of what the customer spent by day. Then someone in the company decides to offer customers a discount when certain numbers are called. Now, if the warehouse had contained every single telephone call made by its customers, the company would be able to find out exactly what this scheme would have cost had it been implemented over the last 12 months. Instead, no data is available and one would either have to guess what the cost might be or postpone the planned new system until sufficient data is available to accurately determine the true cost to the company.

One of the difficult decisions for the designer is to determine at what level data will be stored in the warehouse. Often, storing every transaction, such as in our telephone example, may seem rather excessive, and, because it could easily mean the warehouse grows to many terabytes, there is a temptation to consolidate the data. Managing a terabyte warehouse requires careful and stringently controlled procedures. The bigger the database becomes, the harder it is to manage and query it. Therefore, there is a temptation to aggregate the information.

Since aggregation is a major design decision, the designer would be wise to seek approval from the users of the warehouse before adopting such a strategy. It should also be clearly explained to these users the limitations that are likely to occur due to aggregating the data. With disks being so cheap, hopefully most sites will store all of the data that they require.

2.1.1 Don't use entity-relationship modeling

The typical approach used to construct a transaction-processing system is to construct an entity-relationship (E-R) diagram of the business. This is then ultimately used as the basis for creating the physical database design, because many of the entities in our model become tables in the database. If you have never designed a data warehouse before but are experienced in designing transaction-processing systems, then you will probably think that a data warehouse is no different from any other database and that you can use the same approach.

Unfortunately, that is not the case, and warehouse designers will quickly discover that the entity-relationship model is not really suitable for designing a data warehouse. Leading authorities on the subject, such as Ralph Kimball, advocate using the dimensional model, and we have found this approach to be ideal for a data warehouse.

An entity-relationship diagram can show us, in considerable detail, the interaction between the numerous entities in our system, removing redundancy in the system whenever possible. The result is a very flat view of the enterprise, where hundreds of entities are described, along with their relationships to other entities. While this approach is fine in the transaction-processing world, where we require this level of detail, it is far too complex for the data warehouse. If you ask a database administrator (DBA) if he or she has an entity-relationship diagram, they will probably respond that there was one once, when the system was first designed. But due to its size and the numerous changes that have occurred in the system during its life-time, the entity-relationship diagram has not been updated, and it is now only partially accurate.

If we use a different approach for the data warehouse, one that results in a much simpler picture, it should be very easy to keep it up-to-date and also to give it to end users to help them understand the data warehouse. Another factor to consider is that entity-relationship diagrams tend to result in a normalized database design, where, as in a data warehouse, a denormalized design is often used.

2.1.2 Dimensional modeling

An alternative to using the entity-relationship model is the dimensional model, which is different from the entity-relationship model because it views data from a different perspective. Instead of considering an entity, which represents a thing such as a product or a place and the relationships between those entities, a dimensional model describes data using dimensions and facts, which become actual tables in the database.

Dimensional models, as illustrated in Figure 2.1, despite sometimes looking quite simple, provide a very effective way of holding historical and current data in a form that makes this data accessible to business users and that enable them to make the right business decisions. A dimensional data warehouse can be viewed as containing data that:

  • Has been validated (i.e., no invalid product codes)

  • Is historical (i.e., the last 36 months)

  • Is integrated (the same key is used by all systems)

  • Is easily accessible

Click To expand
Figure 2.1: Dimensional modeling.

2.1.3 Fact table

The fact table, of which there could be more than one, contains factual information, is usually the largest table in the data warehouse, and is typically fast growing. The fact tables are typically where all of the detail data that you want to keep in the data warehouse are stored, such as all of the telephone calls made by a customer or the orders placed by your customer, as shown in Figure 2.1.

Therefore, if a customer made 20 telephone calls, then it is likely that 20 rows will be stored in the fact table for this customer. Consequently, the fact tables will be by far the largest table in the database, possibly containing hundreds of millions of rows in a large data warehouse. If you are unsure as to whether data is factual, it is often numeric and is sometimes a value that may be computed, such as the value of an order or the number of items purchased.

The information contained in the fact table doesn't have to be at the finest level of detail; it could be summarized data, such as total telephone calls made by a customer today. The level at which data is held in the fact table is known as the granularity and is one of the important decisions the ware-house designer must make. In the example described here, the difference in the number of records stored over a 24-month period would be huge. Contrast the storage requirements between a record stored for every telephone call a customer made in a single day compared with a record for every telephone call a customer makes.

When designing a data warehouse, depending on your business, you may find that there are different types of fact tables, such as transaction level, transaction-item level, event based, status, or even summarized data.

2.1.4 Dimension table

When designing using the dimensional model, there may only be one or a small number of fact tables, but there could be many dimension tables. The dimension table can be seen as a reference table to the fact table, where descriptions and more static information about a piece of data are held. For example, product is considered a dimension because, in this table, everything about the product is held, such as full product name, suppliers, and pallet size. In the fact table, there would be a column called "product_key," which is used to retrieve all of the product information from this dimension table.

If you are uncertain as to whether data is a dimension or a fact, ask this question: Is the data relatively static? Typically, dimensions such as a product_id do not change frequently, whereas a fact table would contain the details of the products you had sold. There is also usually an order of magnitude of difference between the number of rows in the fact table, compared with the much fewer rows in the dimension table.

A fact table will contain millions of rows, whereas a dimension table could have only a few rows (e.g., the time dimension could have as few as 52 rows if data was stored weekly for one year). Or a region dimension could contain only 15 rows, if the country had only 15 regions. Dimensions don't have to be small in size, because you could sell 50,000 products or have a customer dimension with 5 million rows. All of these are examples of valid dimensions.

It is hard to say how many dimensions your design will require, but typically there will be less than 20 dimensions and at least four. Therefore, our data warehouse will comprise only a few tables, but it will have huge storage demands because of the number of rows in the fact table.

2.1.5 Warehouse keys

Data in the warehouse will most likely come from a variety of sources, and a product code in one system may not be the same as in another system. Another problem is that when data is being stored over a period of time, keys used in the production system could be reused. Therefore, the designer should seriously consider implementing surrogate keys, so that they have total control over how data is identified within the data ware-house. The conversion of the production key to the data warehouse key will be handled during the ETL process and incurs negligible overhead during data loading. All keys are candidates for being transformed into surrogate keys, and that even includes time keys. Your surrogate keys do not have to be very sophisticated and could simply start at one and increase sequentially using Oracle sequences.. There may also be data storage savings if surrogate keys are implemented.

2.1.6 Normalizing the data warehouse

When it comes to whether the data in the warehouse should be normalized, not everyone agrees on the same approach. Some experts believe that the warehouse should be normalized, while others think that using dimensional normal form is more appropriate.

Dimensional normal form is rather interesting, because it looks like a combination of normalization and denormalization. In Figure 2.2 we see the difference between the two approaches for the store dimension.

Figure 2.2: Normalized versus dimensional normal form.

The normalized case is also referred to as snowflaking. Although Oracle 9i will accept normalized dimensions, take care using this approach. One of the disadvantages is that it may impact performance because more joins will be needed in queries, which will take time. Snowflaking the dimension is a good example of a technique used in transaction-processing systems but is not always appropriate in the data warehouse.

At first glance, the dimensional normal form version appears to be duplicating the data. While this may seem an unacceptable storage over-head, in reality the number of rows in the dimensions is typically very small when compared with the size of the fact table. Therefore, you will probably be surprised to learn that storing these extra data may only cost you a few tens of megabytes. The advantage of this approach is that now only two levels of navigation in the model are required to access information, thus making it easier to construct queries that return data quickly.

2.1.7 Data warehouse or data mart

An alternative to creating one large data warehouse is to create data marts, where a data mart contains a subset of the data in the warehouse. Data marts have the advantage of being focused into an area of the business, so they could contain regional or departmental data. However, care should be taken if the data mart approach is used, because instead of creating one data source, multiple data marts may be easy to manage and ideal for reporting purposes, but trying to integrate those data might be extremely difficult. Thus, the end result could be data marts containing duplicated data that cannot communicate with each other. However, it is not uncommon for an organization to first create data marts and then use those as the basis for creating the enterprise data warehouse.

2.1.8 The Easy Shopping Inc. example

Throughout this book, we will use an example based on a fictitious company called Easy Shopping Inc. It is an organization that has no retail out-lets and sells via its Internet site or via satellite or cable television. In Figure 2.3 we can see our dimensional model for Easy Shopping Inc.

Click To expand
Figure 2.3: Dimensional model for Easy Shopping Inc.

In this example, we have a fact table called "Purchases," where we record every item that our customers purchase. Four dimensions have been defined, where we hold customer, product, time, and details of our daily special offers. Although this may be a simple example for the purposes of this book, even the ones that you create will not be that much more complex. However, you will have more dimensions and many more columns in your fact table.

Warehouse schemas are sometimes called star schemas, and Figure 2.3 is an example of one. The center point is the fact table, and the dimensions sit around the fact table as the points of our star. As per our entity-relationship diagram, once you have drawn the dimensional model, it can easily be translated into a physical database design, since each box represents a table. Although in this book we refer to fact and dimension tables, inside the Oracle database they are all tables and are treated as such. However, before you jump in and create the physical database from this dimensional model, there are a few more decisions to make before the design is complete.


Other design considerations

2.2 Other design considerations

When you are constructing a data warehouse, it is easy to become focused on ensuring that queries are processed quickly. However, blindly following this approach could easily result in a database that is difficult to manage or use.

2.2.1 Design to manage

It's no good building a warehouse that answers all questions in under a minute if the data inside it is at risk because the database cannot easily be backed up. Therefore, always identify the crucial management tasks and determine if they can be performed easily using this design when designing a database. In Chapter 7 we will discuss in more detail the management tasks for a data warehouse, but let us briefly review some of those tasks and see how they affect the design. Some of the important management tasks include:

  • Backup

  • Loading new data

  • Aggregating new data

  • Data maintenance activities such as indexing and archiving

All databases should be backed up regularly, and Oracle 9i has the RMAN utility, which allows on-line backups and incremental backups of the data that have changed. On the surface, backup may seem a trivial task, but backing up a terabyte warehouse takes time, even if it's an on-line backup. Therefore, the designer should carefully consider the tablespaces (explained in Chapter 3), where the data is stored to make tablespace backups of read-only tablespaces, or use partitioning (explained in Chapter 3) so that a full backup can be taken by running parallel backup tasks; you could even consider using Data Guard to back up the data warehouse.

Full database backups are likely to be a luxury in very large warehouses; therefore, you should design the warehouse to allow incremental backups to be taken that contain only the changes to the warehouse data. Due to the huge volumes of data in the warehouse, any operation that can be performed in parallel will significantly reduce the time required to complete the task, especially if there are many parallel processes running concurrently. Most of the new data will have to be stored in the fact table; there-fore, during the design phase, the designer should ascertain when and how much data is going to be loaded. Then calculate the anticipated load time, and if it cannot all be loaded in the available time, techniques such as partitioning the fact table so that the data could be loaded in parallel should be considered.

When the data warehouse is being tested by the design team, they should not concentrate only on performance testing, which will enable them to advise how long loads will take and discuss with the operations department how backups will be performed and how much time they'll need.

In Chapter 4, we will see how summary management, which was introduced in Oracle 8i, can be used to maintain aggregated data. One of the performance techniques widely used in the warehouse is to create summary tables of preaggregated data, known in Oracle 9i as a materialized view. Then a query transparently reads the materialized view instead of having to read all of the detail data. Hence, the performance improvements can be enormous, depending on the reduction in rows between the detail and the materialized view.

Unfortunately, we get nothing in this world for free and materialized views have to be maintained. This can involve considerable I/O, depending on how many new records are added and whether the materialized view is created completely from the beginning or if only new data is added. If many materialized views are defined and they are all refreshed at the same time, then consideration should be given to placing the materialized views in different tablespaces on different disks. Failure to do this will result in all I/O occurring on the same disk, thus slowing the refresh process considerably. This may be an even more important consideration if the refresh operations are to be performed in parallel.

2.2.2 Design for performance

As we will see in Chapters 3 and 4, there are various techniques that can be employed by the designer to improve query performance. Some will involve how queries are constructed, but many are actually in the database design. For example, all databases benefit from indexes, and a data warehouse is no exception. Therefore, do not forget to decide which type, where, and how much space is to be allocated for indexes. In a data warehouse, the designer does not have to worry about many users inserting new entries into the index and the associated problems that can result. Instead, the designer is now concerned with the time that is required to maintain or build an index. For instance, recreating an index on a fact table with 100 million rows will take more than a few minutes to complete!

Oracle 9i offers different types of indexes, and the designer should select the one that is most appropriate, which could mean that the design will contain many bitmapped indexes. Different types of queries in a data ware-house will use different access methods to the data that benefit from the different index types (e.g., a star transformation uses a bitmapped index).

Physical placement of the data is another important consideration, especially if it is used in conjunction with partitioning and parallel operations. If data is physically located on different disks, then queries or tasks can be performed that do not saturate the I/O limits on a specific disk drive.

One of the significant performance gains is when materialized views, which are described in Chapter 4, are used. Since they have to be created and maintained, once again space must be reserved for this data, and the improvement in query response time must be balanced against the time required to maintain this data.

New data for a warehouse often arrive in batches. Hopefully, they will be loaded into the database when it is not in use, but this cannot be guaranteed. Therefore, if, during your investigations of the proposed system, you discover that data will be loaded into the warehouse while it is in use, review techniques that allow you to insert data into an area different from the one being used by the users (e.g., partitioning).

Another consideration is whether the fact table is likely to be updated. With so many records in the fact table, this could have a significant impact on performance; therefore, procedures may have to be put into place to stop unauthorized updates to the fact table.


Implementing the design

2.3 Implementing the design

Once we are satisfied with the database design, it is time to physically create our database. Initially, one should create a small-scale version of the database and test design ideas here before building the full-size production system. There are various tools available to the designer to help create the warehouse; these will be discussed in other chapters. Here we will see how to create our database using nothing more sophisticated than SQL.

Database designers often prefer to create a script file containing the SQL commands to create the database, and this is perfectly acceptable. An alternative approach is to use the graphical user interface (GUI) tools in Oracle Enterprise Manager, which will be illustrated here.

Hint:

If the SQL is complex, use these GUI tools to create the SQL, and then paste the SQL into your text file.

In this section, we will walk through the various stages required to create our data warehouse. First we will see how to create the actual database and then learn how to create the tablespaces and data files where the actual data is stored. That will be followed by illustrations of how to create the tables and a brief introduction on creating the indexes and partitions. Finally, we will discuss materialized view creation and how to secure the objects in the data warehouse that we have just created.

2.3.1 Single database or many?

There was a time when the data warehouse was created in its own database. However, times are changing, and now some companies prefer to have a single database that contains all systems.

There are pros and cons for each approach, and whether you choose a single database or multiple databases will depend upon your business requirements. Creating a new database is not a difficult job, and the best approach is to use the GUI tool, Oracle Database Configuration Assistant. Once the database has been created, you can then add your own data files and tablespaces using Oracle Enterprise Manager Console GUI or SQL scripts.

A database can be created directly from SQL, but, if this approach is used, then it should be performed with care, because you will need to run a number of script files that are required by Oracle 9i. If you use the GUI, this work is done automatically.

2.3.2 Naming conventions

Before anything is created in the database, the naming conventions used for all database objects, such as data files, tablespaces, and table and column names, should be reviewed. Depending on the tools available to your end users, they may actually see these table and column names. Therefore, if they do not have sensible names, these end users, who are generally not computer literate, could be very confused.

In our Easy Shopping Inc. example, there is a column in the fact table called "time_key." Now, for people familiar with databases, it is obvious what that field contains, but to our end users of the warehouse it means nothing. Therefore, in this instance, a better column name might be "date_time_of_purchase." The warehouse designer should also remember that if end users will be using the warehouse, a more English-like column name should be provided.

So far we have not discussed the topic of metadata, but in a data ware-house metadata is very important. There should be one definition of a data item, which in the ideal world would have only one set of values. For example, a region code is supposed to be a three alphanumeric code, and it is in all systems except one, where it is defined as a number.

These are some of the challenges for the team responsible for loading the data into the warehouse, they are discussed further in Chapter 5, where we will apply all the necessary conversions to the data to ensure that, when this data is in the warehouse, all values are the same.

2.3.3 Oracle Database Configuration Assistant

Oracle 9i includes a number of GUI tools to assist with managing the database. A very useful one is the Oracle Database Configuration Assistant, from which you can create, delete, and modify a database or manage the templates used to create a database. This tool runs standalone (look for DBCA in the %ORACLE_HOME%/bin directory), and it does not require Oracle Enterprise Manager Server to run. In Oracle 9i it comes with three preconfigured databases, and selecting one of these will significantly reduce the time it takes to create the database; otherwise, you can create a new database.

Once you have decided which type of database is required, it must be given a name. As shown in Figure 2.4, we have called the database EASYDW.

Click To expand
Figure 2.4: Oracle Database Configuration Assistant—naming the database.

The following steps will vary, depending on whether a preconfigured or new database is built.

Using a preconfigured database

There are three types of preconfigured database supplied with Oracle 9i:

  • Data warehouse

  • General purpose

  • Transaction processing

If you are unsure which one is suitable for your environment, the physical attributes of the database and installed components can be seen by clicking on the Show Details button, which will show the template being used.

There are only a few screens requiring input when a preconfigured database is used. One of the first is defining some of the parameters used to configure the database. An Oracle 9i database has a number of components that can be configured. In Figure 2.5 we see step 5 of seven steps from the Database Configuration Assistant, where we can define the parameters for the respective areas by clicking on the appropriate tabs, as follows:

  • Memory

  • Character sets

  • Memory used during sort operations

  • File locations for initialization and trace files

  • Archiving

Click To expand
Figure 2.5: Oracle Database Configuration Assistant—initialization parameters.

The default parameters may be suitable for your environment. However, if you are building a small database to evaluate some features and the memory available to you is limited, then click on the Custom option and amend the memory being allocated. You can see the effect of your changes instantly by monitoring the Total Memory for Oracle value.

Hint:

If a parameter is set too low, you will be advised and given an opportunity to increase its value.

One of the buttons shown in Figure 2.4 is File Location Variables. When the database is created, the location of all the files that comprise the database—that is, the data files, control files, initialization, redo log, and archiving files—are located according to the values of these parameters that you can configure for each database, as illustrated in Figure 2.6. Also at this time you can define your own variables and use this in the definition of your database.

Click To expand
Figure 2.6: Oracle Database Configuration Assistant—file location variables.

When using one of the preconfigured databases, you cannot control the size of the database, only the location of the files, which is the screen shown in Figure 2.6. By default all of the files are placed in one directory; therefore, if you want them on different disks, now is the time to specify the new location.

All the information needed to build the database has now been defined, and you can now create the database or save this definition as a template for another time.

Using this approach your database is now ready for creation, and you can monitor its progress as it is being built. Once the build has finished, it is available for use.

Creating a new database using Oracle Enterprise Manager

Although creating a new database takes considerably longer than using one of the preconfigured ones, it does offer more flexibility in what is created, and, consequently, the steps to complete the task are different.

Once the database has been named, as shown in Figure 2.4, you will be asked which database features you require inside your customized database. In Figure 2.7 we can see that there are a number of features that we do not have to install. For example, here we have decided not to include Oracle Spatial, but we do want Oracle Data Mining.

Click To expand
Figure 2.7: Oracle Database Configuration Assistant—database features.
Hint:

Database creation time can be significantly reduced by only installing the features you require.

Along with these features there are some sample schemas that can be installed, where examples are provided to help illustrate the functionality available in these features.

Next, the parameters we saw in Figure 2.5 must be defined; default values will be supplied, or the ones shown can be amended as required. When the DB Sizing tab is selected, there will also be the option to specify the database block size, which defines the lowest level of granularity and control you have concerning the allocation of space in the warehouse. It is recommended that your data warehouse be created with a large block size, such as 16,384 bytes, which means that you can store more records of the same type together in a single block, thus helping to reduce our I/O demands.

The next screen, which is shown in Figure 2.8, is where all of the files that comprise the database can be created, removed, or modified.

Click To expand
Figure 2.8: Oracle Database Configuration Assistant—database storage.

Using this screen, tablespaces can be added by clicking on the Add button. Here we can see that a new tablespace called SUMMARY has been added. Some tablespaces will automatically exist, and expanding the navigation window on the left and then selecting that item can modify the size and characteristics of these tablespaces.

Datafiles, rollback segments, and Redo Log Groups can also be defined at this stage by pressing the Add button.

The creation of the database is almost ready to begin. Now you can create the database; save this configuration as a template to use again later, and, optionally, request the creation of a SQL script so that you can create this database later.

If you request a SQL script, then a number of SQL files will be stored at a location you specify and there will be one master script, named using the database name specified in Figure 2.4, that will run all of these scripts.

If you decide to create the database now via the GUI, you can see how the creation of the database is progressing. It was mentioned earlier that during database creation a number of scripts are run against the database to provide certain functionality. Creating a database will take many minutes, so once it starts, take a short break, or do something else for a little while.

2.3.4 Which schema?

Now that we have a database, all objects defined in an Oracle database must reside inside a schema, a logical structure that describes a collection of objects. Therefore, before any tables are created, you should decide how many schemas you require. Since a data warehouse may contain only a few tables, it is probably a good idea to keep them all in one schema.

However, you may prefer to create multiple schemas by subject area, but this approach will also increase the effort required to manage the database to make the information in the tables available. This is because there will be extra tasks to perform, such as granting access to tables and defining synonyms.

Hint:

It is very important to make this decision at the outset of the design, because the schema name plays an integral part in the naming convention used to retrieve information from the database.

A schema object is created every time a database user is defined. A user is created via the SQL CREATE USER statement or from within Oracle Enterprise Manager by selecting User from the object list and completing the window shown in Figure 2.9.

Click To expand
Figure 2.9: Oracle Enterprise Manager—create a user.

In our Easy Shopping Inc. example, we have decided to create a user called EASYDW. Here we can specify how the user will be authenticated, and we have selected the default mode of By Password. This means a password must be specified and used for all subsequent database access. If, in the future, this password needs to be changed, it can be done from within Oracle Enterprise Manager, using the screen shown in Figure 2.9 or via SQL.

We must also select a default area where objects created by this user, such as tables and indexes, will reside. In a well-designed database, this is not an issue, because every object will explicitly state in which tablespace it must be stored. There are a number of other options that can be specified for a user, such as privileges. It is important to set these; otherwise, you will not be able to retrieve data. As we progress through this book, you will be advised about which privileges are required when a topic is discussed.

When the user is created, a schema is automatically created with that user name. However, you will not be able to see the schema name until the first object, such as a table, is defined for that user.

The schema name is very important, because it is used to fully qualify an object in the database. For example, we could have a table called TIME in the EASYDW schema and also in our ORDERS schema. To advise the optimizer which table you wish to retrieve data from, you specify the table name as:

schema name. table name

Therefore, to retrieve all the records in our time dimension, the fully qualified table name would be:

SELECT * FROM easydw.time;

When Oracle Enterprise Manager first starts, the standard navigation window is displayed in the left frame. If you click on Schema, a list of all of the users defined in this database is displayed, as illustrated in Figure 2.10.

Click To expand
Figure 2.10: Oracle Enterprise Manager Console—schemas.

You can create as many users of the database as you require, but it is recommended that only one of them should be used for the purpose of creating objects, such as tables and indexes. Therefore, when the DBA connects as user EASYDW, all the tables and indexes created will reside here, as illustrated in Figure 2.10. In section 2.3.11, we will discuss enabling privileges for a user.

2.3.5 Data files and tablespaces

Once the database has been created, you can add your own data files and tablespaces. By default, you will find a number of data files; on a Windows NT system, they will be located in:

\<Oracle_Home>\Oradata\<database name>\

and will comprise two control files, index area, system space, rollback, and user area.

Inside these tablespaces is where objects are stored in the Oracle database, and a tablespace can have one or more data files associated with it. Therefore, the tablespace is the logical name that is used within the database schema to specify where objects must reside. An object is then actually stored in one or more data files, and part of the tablespace definition is their physical location and size.

In our Easy Shopping Inc. example, we have decided to implement the following tablespaces:

  • Dimensions—for all dimension data

  • Default area that users are assigned by default

  • Summary—for the materialized views we will create

  • Purchases_month_year for our partitioned fact table

  • Indx tablespace—for indexes

  • Temp area—for temporary space

In this example, we will create only one data file per tablespace, but, of course, you can create more if required. Also, the files shown here will be very small, and in the real world, they could be extremely large.

These tablespaces and their associated data files can be created either directly from SQL or by using the Oracle Enterprise Manager Console tool, which is part of Oracle Enterprise Manager, illustrated here in Figure 2.11.

Click To expand
Figure 2.11: Oracle Enterprise Manager Console—tablespaces.

When you are managing a database using Oracle Enterprise Manager, it means that you no longer have to keep querying the metadata to find out the state and information on objects in your database, because the information is already available.

In Figure 2.11, we see the EASYDW database; we have then selected the database EASYDW, then storage, and then tablespaces, so in the right-hand window we see a list of tablespaces, their size, current state, and space used.

If you prefer to create your tablespace directly from SQL, then you can use the SQL*Plus utility. If you are unsure of the SQL required to perform a task, you can click on the Show SQL button in most of the GUIs, and a window appears at the bottom of the display containing the SQL. In Figure 2.12, we can see the SQL to create the dimension tablespace.

Click To expand
Figure 2.12: Oracle Enterprise Manager Console—dimension.

Creating a tablespace using Oracle Enterprise Manager is very easy. Simply select the database, then Storage followed by Tablespace and then right-click on the mouse and select Create. The screen shown in Figure 2.12 appears, where you specify all of the attributes of this tablespace.

The first step is to enter the name of the tablespace. It is wise to choose sensible names, because you will be using these constantly throughout the schema, and it helps if they mean something to you. For example, the tablespace called DIMENSIONS will be used to hold the dimension tables, whereas the purchases made in January are held in a tablespace called PURCHASES_JAN2002. Another favorite approach is to suffix all data areas with a "D" and indexes with an "X," to identify the type of data that is held in that tablespace.

To specify the location of the data file, click in the File Directory box to change the default location. The same approach is used to change the name of the file and its size. Click on the Storage tab to display the page where you can set the file to automatic allocation. Some designers do not like files that automatically extend; however, many do, because it means that a task will not fail because there is no space left in the database. Instead, the data file will automatically extend itself, and you can control how large those extents are. Of course, when the disk is full, autoextend will fail.

2.3.6 Creating the fact and dimension tables

Now that we have a database and the tablespaces and users are defined, we are ready to create the fact and dimension tables. The fact and dimension tables are created as if they were any ordinary tables inside the database. Therefore, all the options one would specify on a table, such as the initial and subsequent extent size, can be specified. Although we call them "fact" and "dimension" tables, they are no different from all of the other tables in the database.

When defining the fact table, carefully select the column data types, because selecting one that occupies too much space, when your fact table contains hundreds of millions of rows, will create a considerable waste of disk space.

Tables are created using the SQL CREATE TABLE command, but we will now see how to create tables quickly using Oracle Enterprise Manager. For the moment, a dimension table is defined as if it were any other table in the database. In Chapter 4, we will see how to define an actual dimension object, which will be based on the table that we created here. In fact, the dimension table created at this stage is a prerequisite for creating a dimension object.

When Oracle Enterprise Manager Console is started, if you select Object, Create, Table, Create, the Table Wizard will start, the first screen is shown in Figure 2.13.

Click To expand
Figure 2.13: Oracle Enterprise Manager Console—Table Wizard.

In Figure 2.13, you specify the name of the table, the schema in which it will reside, and the tablespace for this table. Clicking on the Next button will display Figure 2.14, which is where you define the columns that are to appear in the table.

Click To expand
Figure 2.14: Oracle Enterprise Manager Console—table columns.

This is a really nice screen for quickly creating the table. All you have to do is enter the column name, select the data type and its size, and then press the Add button, and the column appears in the list on the left of the window. Define each column using this approach, and don't click on the Next button until you have defined all of the columns in the table.

If you do click on the Next button by mistake, you can return to this screen by clicking on the Back button. Once all of the columns in the table have been defined, the following screens will allow you to create the constraints.

2.3.7 Constraints

The job of the constraint is to ensure that all data conform to its rules, such as a value corresponds to a specified range of values via a CHECK constraint. If a primary key is defined, then this guarantees that the value is unique. A foreign key will ensure that all values correspond to one of the primary keys in another table.

Mention constraints to designers, and they will probably tell you that they do not want them in the database because they are an overhead, especially when new data is being loaded. It is highly recommended that you implement at least primary-and foreign-key constraints, especially if you wish to use the Summary Management feature described in Chapter 4. One of the interesting aspects of Summary Management is the ability to rewrite a query to use a materialized view. If you have defined constraints in your database, then it will be possible to do some complex forms of query rewrite.

In a data warehouse, there is often concern that because the data can come from many sources, it may not be as "clean" as normal data, and, therefore, constraints may fail. Although this is a valid concern, clean data should always be stored in the warehouse to ensure that accurate results are returned.

Another argument put forward for not implementing constraints is that validating every record when it is first inserted into the database imposes a considerable burden on the load operation. Therefore, it takes considerably longer than via a standard load, and if the loading window is very small, then one way to reduce the time is not to have constraints. However, the ETL stage can also be used to programmatically validate the constraint candidates, resulting in them being disabled as novalidate.

In Oracle 9i, many of these concerns can be overcome thanks to some options on the constraint:

  • ENABLE NOVALIDATE

  • DISABLE NOVALIDATE

If you are still concerned about the overhead of having constraints and worried that your data isn't clean enough to get past the constraint checks, you can use the ENABLE NOVALIDATE clause, which turns on a constraint and applies it against all new inserts and updates, but it doesn't check existing records. It is enabled immediately, but you should be aware that incorrect results could be returned if existing rows in the table have violated the constraint.

By using the ENABLE NOVALIDATE clause as illustrated in the following code segment, we can turn on the constraint SYS_C001136 without having to validate all of the data.

SQL> ALTER TABLE todays_special_offers
   ENABLE NOVALIDATE CONSTRAINT SYS_C001136;

If we know or want to assume that the data is clean, we can just turn on the constraint immediately without incurring any overhead. Using this approach, the database doesn't spend time validating the constraint against all the rows in the table, but it does mean that the designer had better be certain that the data is clean.

Hint:

Use sensible constraint names, such as customer_pkey, which will mean so much more than SYS_C001136.

Also, before data is loaded, the constraints can be quickly disabled using the DISABLE clause, as shown in the following code segment:

SQL> ALTER TABLE todays_special_offers
   DISABLE CONSTRAINT SYS_C001136;

At the time of writing, Oracle Enterprise Manager does not support the ENABLE/DISABLE NOVALIDATE clause; therefore, you will have to enter this manually in SQL. But you can ENABLE/DISABLE the constraint within OEM.

There is an additional clause called RELY, which is required by the Summary Management feature. This clause tells the optimizer that you can rely on the accuracy of the constraint. An example of using the RELY clause is shown in the following code segment on a constraint in the "todays_special_offers" table.

SQL> ALTER TABLE purchases MODIFY CONSTRAINT special_offer RELY;

You can check the constraints that have been defined in your database using the Schema Manager. Select the table of interest and click on the Constraints tab, as shown in Figure 2.15.

Click To expand
Figure 2.15: Oracle Enterprise Manager Console—constraints.

In Figure 2.16, we see how to create a primary key within the Table Wizard by clicking on the column for the key. Not every table in our data ware-house, such as the fact table, will have a primary key, but some of the columns may require that they are not null.

Click To expand
Figure 2.16: Table Wizard—specifying the primary key.

Clicking on the Next button in the Create Table Wizard displays Figure 2.17. For each column in the table, click on the Radio button to specify whether it may be null or unique, and then click on the column names in the list on the left side of the screen to move back and forth between the columns in the table.

Click To expand
Figure 2.17: Oracle Enterprise Manager—null constraint.

Clicking on the Next button again will display Figure 2.18, where you can specify whether any columns are foreign keys.

Click To expand
Figure 2.18: Oracle Enterprise Manager—foreign key.

On pressing the Next button, Figure 2.19 appears, where you can specify check constraints. A check constraint enables you to guarantee the values that a column will take. In our example in Figure 2.19, we have specified that the column DAY may take only the values between 1 and 31. Therefore, before a value is stored in this database, a check is automatically made by the database server to see if the column takes one of these values.

Click To expand
Figure 2.19: Oracle Enterprise Manager—check constraints.

An alternative to using the GUI to obtain information from Oracle 9i is to query the many system tables that are available; they provide a wealth of information about the state of your database and objects. The following code illustrates the information held about constraints in the table USER_CONSTRAINTS.

SQL> DESCRIBE user_constraints
 Name                                 Null?    Type
 -----------------------------------  -------- ------------
 OWNER                                NOT NULL VARCHAR2(30)
 CONSTRAINT_NAME                      NOT NULL VARCHAR2(30)
 CONSTRAINT_TYPE                               VARCHAR2(1)
 TABLE_NAME                           NOT NULL VARCHAR2(30)
 SEARCH_CONDITION                              LONG
 R_OWNER                                       VARCHAR2(30)
 R_CONSTRAINT_NAME                             VARCHAR2(30)
 DELETE_RULE                                   VARCHAR2(9)
 STATUS                                        VARCHAR2(8)
 DEFERRABLE                                    VARCHAR2(14)
 DEFERRED                                      VARCHAR2(9)
 VALIDATED                                     VARCHAR2(13)
 GENERATED                                     VARCHAR2(14)
 BAD                                           VARCHAR2(3)
 RELY                                          VARCHAR2(4)
 LAST_CHANGE                                   DATE
 INDEX_OWNER                                   VARCHAR2(30)
 INDEX_NAME                                    VARCHAR2(30)
 INVALID                                       VARCHAR2(7)
 VIEW_RELATED                                  VARCHAR2(14)

Every constraint that you define for this schema will be recorded in this table. All of the system tables provided by Oracle will be prefixed either ALL_, USER_, or DBA_.

Therefore, to see on which constraints you have used the RELY clause, use the query shown in the following code:

SQL> SELECT constraint_name, table_name, rely FROM all_constraints
     WHERE OWNER = 'EASYDW';
CONSTRAINT_NAME                TABLE_NAME                     RELY
------------------------------ ------------------------------ ----
COST_PRICE_NOT_NULL            PRODUCT
FK_CUSTOMER_ID                 PURCHASES
FK_PRODUCT_ID                  PURCHASES
FK_TIME                        PURCHASES
NOT_NULL_CUSTOMER_ID           PURCHASES
NOT_NULL_PRODUCT_ID            PURCHASES
NOT_NULL_TIME                  PURCHASES
PK_CUSTOMER                    CUSTOMER
PK_PRODUCT                     PRODUCT
PK_SPECIALS                    TODAYS_SPECIAL_OFFERS
PK_TIME                        TIME
PUBLIC_HOLIDAY                 TIME
SELL_PRICE_NOT_NULL            PRODUCT
SHIPPING_CHARGE_NOT_NULL       PRODUCT
SPECIAL_OFFER                  PURCHASES                      RELY

Here we can see that the constraint SPECIAL_OFFER has the RELY clause enabled, whereas constraint SHIPPING_CHARGE_NOT_NULL does not. In Figure 2.20, we can see the constraints that have been defined on the table PURCHASES.

Click To expand
Figure 2.20: Oracle Enterprise Manager—viewing constraints.

At the time of writing, you can see in Oracle Enterprise Manager whether the constraint has been enabled or disabled, but you cannot see if it has been validated. This information can be determined by querying one of the constraint system tables, such as USER_CONSTRAINTS. To see which constraints have been enabled using the NOVALIDATE clause, use the query shown in the following code:

SQL> SELECT constraint_name, validated FROM user_constraints
  WHERE OWNER = 'EASYDW';
CONSTRAINT_NAME                VALIDATED
------------------------------ -------------
PK_CUSTOMER                    VALIDATED
COST_PRICE_NOT_NULL            VALIDATED
SELL_PRICE_NOT_NULL            VALIDATED
SHIPPING_CHARGE_NOT_NULL       VALIDATED

The definition of the table is almost complete. We have defined the columns and the constraints that are required, and we could press the Finish button, but there are two more categories of information that the Table Wizard requests—storage and partitioning—which will be described later.

The final screen in the Create Table Wizard is shown in Figure 2.21, where we can see the SQL that will create our table. If you are happy with the table definition, then press the Finish button; otherwise, press the Back key, and amend the entry accordingly.

Click To expand
Figure 2.21: Table Wizard— final screen.

When you click on the Finish button, the table is created, and you are now ready to create the next table. Hopefully, you will agree that this is a very easy way to create a table, and, since a data warehouse probably only has a few tables, you may prefer to use this friendly approach as opposed to writing SQL commands, where you will probably make many syntax errors, which you will have to correct.

2.3.8 Indexes

A data warehouse is likely to contain a number of indexes, and, as with any other database, the designer must choose the indexes that are most suitable. Oracle offers several different types of indexes, but the ones that will be of interest to the designer are as follows:

  • B*tree index

  • Bitmapped index

  • Bitmapped join index

Indexes should be selected carefully, and the various options and reasoning behind certain choices will be described in detail in Chapter 3. Please consult this chapter, because there you will learn whether to select a global or local index and how to partition it if required.

To set the scene for this section, a bitmapped index is ideally suited to the data warehouse environment when you want to index a column that takes only a few values. For example, suppose we wanted to index the column PUBLIC_HOLIDAY, which has only two values: "Y" or "N." A bitmapped index, which is described in more detail in Chapter 3, will store this information in an extremely compact fashion; this also has additional benefits for the way that the Oracle optimizer accesses the data for typical warehouse queries.

Although indexes can easily be dropped and created, due to the time required to create them, especially on a fact table with millions of rows, careful planning at the outset of the project will ensure that you won't have to spend a lot of time creating the index. An index can be created by using either the SQL CREATE INDEX command or Oracle Enterprise Manager. (See Figure 2.22.)

Click To expand
Figure 2.22: Oracle Enterprise Manager—indexes.

2.3.9 Partitioning

Partitioning data is a design technique that is very important in the ware-house because it provides a means of managing large amounts of data and controlling their placement on the disks. Rather than place all of the data from a table in one tablespace, partitioning enables us to place the data in many tablespaces. To determine in which tablespace data is stored, a partition key is selected, such as time_key, illustrated in Figure 2.23.

Click To expand
Figure 2.23: Partitioning.

In this example, we are partitioning by month, so January's data go in one partition, and February's in another, and so forth. You must select the partition key carefully, although a common one is by time. Therefore, you could partition data by month, and then each month would reside in its own tablespace. This helps result in a more manageable partition, and it also has the advantage that if you ever had to archive the data, it would be as simple as dropping a partition. Of course, don't forget to back up the partition before you drop it! This is a very quick process and doesn't invalidate any of the data that are already in the fact table.

Oracle 9i provides several different types of partitioning techniques, and, after reading the partitioning section in Chapter 3 (section 3.3), you can pick the one that is most appropriate for your data warehouse.

When a table is being created using the Table Wizard, there are five screens that allow you to range partition a table. The first of those screens is shown in Figure 2.24. If you want to use any of the other partitioning methods, such as composite, then the table will have to be partitioned manually.

Click To expand
Figure 2.24: Table Wizard—partitioning screen.

The other four screens allow you to specify the partitioning criteria, illustrated in Figure 2.25, where you can easily define your range partition.

Click To expand
Figure 2.25: Table Wizard—partitioning criteria.

This is followed by a screen where you specify the tablespace used for each table partition, the columns to be used on the indexed partition, and the tablespace for each of those index partitions.

2.3.10 Materialized views

We have already seen that a data warehouse or data mart can hold a huge number of records in the fact table. Even if we had the fastest machine in the world and could cache some of the data warehouse in memory, the time required to respond to queries could be days—and it would certainly be minutes or hours.

To overcome this problem, warehouse designers use the technique of creating summaries, a summary being a preaggregated table of results, which Oracle calls a materialized view. For example, suppose you always query the number of purchases of today's special offer by day. Rather than compute those results every time, a materialized view is created that contains the required information. Then, whenever you make this query, instead of querying the fact table, you query the materialized view instead.

Although it partially defeats the object of a warehouse, where you make unknown queries to the database, it is fair to say that quite a few queries upon the warehouse are well known. If we can improve the response time on those queries, our users will be very grateful.

Oracle 9i includes a specific Summary Management Component, which will enable you to create materialized views rather than ordinary tables; then the optimizer will transparently rewrite your query to use the materialized view. This feature is described in detail in Chapter 4. At this stage of the design, if you can identify any queries that would lend themselves to being created as materialized views, they should be recorded now for subsequent creation. Some examples of the materialized views that we might create for our Easy Shopping Inc. example are as follows:

  • Sum of sales by product by day

  • Count of products sold by day

  • Sum of sales by week

  • Profit by product by day

The number of materialized views you expect to create will determine how many tablespaces and data files should be defined for this data ware-house.

Hint:

It's not necessary to create a materialized view for every possible combination; this will be explained in Chapter 4.

2.3.11 Security

One should not forget that some data in the warehouse could be very sensitive, and, therefore, for a variety of reasons, you may not wish for all of your staff to have access to this data. Oracle 9i provides various types of security that prevents users from changing the objects inside the database and accessing data.

Object privileges

Object privileges can be placed on the following types of objects:

  • PL/SQL modules

  • Queues

  • Sequences

  • Synonyms

  • Tables

  • Types

  • Views

You will most likely place security on tables and views, by stating whether a user can select, insert, and update the data, along with a number of other options. If you decide to create a number of users, then always ensure that sufficient privileges have been allocated so that everyone can read the data. This can be achieved by using either the SQL statement, GRANT SELECT ON command, or the GRANT SELECT ANY TABLE command, or these privileges can be allocated directly to the user name using Oracle Enterprise Manager, within the security section.

In Figure 2.26, we are giving the user EASYDW the rights to ALTER, INSERT, SELECT, and UPDATE the CUSTOMER table in the EASYDW schema. Simply repeat this process for all users with the tables they are allowed to access.

Click To expand
Figure 2.26: Oracle Enterprise Manager—object security.

Role

Alternatively, you could create a role. Then you assign all of the privileges to the role, and the role is assigned to the user. This is the preferred approach—especially if you create many users—because then you can create roles for the different job levels, and each user is granted one of those roles instead of assigning the privileges individually. Using the role approach, you reduce the likelihood of users accidentally being given access to data that they shouldn't have access to. It is also quicker if many users have to be defined. You can either use existing roles or create your own role using the SQL CREATE ROLE statement.

System privileges

Most of the time will be spent granting object privileges to users and maybe creating roles. However, some users will require system privileges. A system privilege is one that gives you the right to perform high-level tasks such as creating or dropping a table. Since you will not want to grant this right to many users, you shouldn't have to spend too much time giving system privileges.

The method used to define a system privilege is the same as for an object privilege, except this time system privileges are chosen. In a data warehouse, there are some specific system privileges that you may want to grant to specific users, such as CREATE DIMENSION and CREATE MATERIALIZED VIEW. In this instance, you would only be granting this privilege to users who would create dimensions and materialized views.

2.3.12 Using the PARALLEL clause

Our database is almost complete, but there is one other important feature that is available in Oracle 9i that should be mentioned, and that is the PAR-ALLEL clause. A number of the statements shown here can be executed in parallel, and the use of this technique is very important in a data warehouse because it can significantly improve statement execution time. Parallel operations are available on table scans, sorts, joins, aggregates, and some table and index operations.

If you are using a symmetric multiprocessor or a massively parallel system, then serious consideration should be given to using the PARALLEL clause. When specified, a statement, if eligible for parallel processing, will be decomposed into a number of parallel threads, and Oracle 9i will perform the job using parallel tasks and coordinate their running and the results. Therefore, all the user has to do is include the clause, and Oracle 9i does the rest.

The PARALLEL clause will expect you to specify the number of parallel processes to use. Choose this value carefully. Some testing may be required to determine the optimum value. For example, the following clause could be added to the CREATE TABLE statement for our purchases fact table:

PARALLEL (DEGREE 2)

This would mean that any operations on this table should be done using two server processes if they can be executed in parallel.

Data may also be loaded in parallel. The SQL*Loader facility, which we will discuss later, allows you to request parallel operations. Obviously, this can significantly improve the time required to store data; however, you should be aware of the possible fragmentation of your data that could occur. Therefore, it is suggested that when data is being loaded in parallel, you specify the number of parallel operations to be equivalent to the number of data files available for that tablespace. Therefore, referring to our Easy Shopping Inc. example, if we decided to specify a value of:

PARALLEL (DEGREE 3)

on our purchases table, there should be three data files defined for every tablespace.


Testing the design

2.4 Testing the design

None of us would write an application and send it live without testing it first. But it is amazing how many database designs are constructed and then unleashed on unsuspecting users. Your data warehouse is no different, especially since the business is relying on it for important information. Therefore, it is very important that all aspects of the design and processes are thoroughly tested prior to production release.

It is suggested that you initially load a small percentage of the data into the warehouse, and then test the following areas:

  • Time required to load the data

  • Data cleansing and transformation

  • Query response times

  • Summary data needs

  • Time required for management tasks

If you are building a terabyte-size warehouse, it is recommended that you repeat this process again with even more data in the warehouse, just in case there are any unexpected problems dealing with this volume of data.

Problems identified during testing are much easier to fix than trying to resolve them once the warehouse has gone live. A phased implementation to the user base is another way to test the warehouse, if you don't want to wait until all of the testing is complete.

One very important point to remember is that due to the size of the data warehouse, it will not only take much longer to load the data, but it is also unlikely that queries will complete quickly. Therefore, the entire testing process will take much longer than, say, a traditional OLTP database.


The schema for Easy Shopping Inc.

2.5 The schema for Easy Shopping Inc.

We have seen in this chapter how to create our database using the GUI tools, but many readers may prefer to create the database directly from SQL. The SQL to achieve this is shown in the following code, but it assumes that the database has already been created.

The example shown here has been created for a Windows 2000 system; a simple edit of the file specs is all that is required for a different platform. Also note that the file sizes here are very small compared with what you would use in a production environment.

2.5.1 Creating the tablespaces and data files

The first step is to connect to the database using a powerful user name:

connect system/manager;

The next step is to create the tablespaces where the data will reside and their associated data files:

-- Create Tablespace for Undo Area
CREATE TABLESPACE easy_undo
 datafile 'D:\EASYDW\easy_undo.f'
 size 8m reuse autoextend on default storage
 (initial 16k next 16k pctincrease 0 maxextents unlimited);

-- Temporary Tablespace
CREATE TEMPORARY TABLESPACE easy_temp
 tempfile 'D:\EASYDW\easy_temp.f'
 size 10m reuse autoextend on next 16k ;

-- Tablespace to store Materialized Views
CREATE TABLESPACE mview
 datafile 'D:\EASYDW\easy_mview.f'
 size 6m reuse autoextend on default storage
 (initial 16k next 16k pctincrease 0 maxextents unlimited);

-- Tablespace for Dimensions
CREATE TABLESPACE easy_dim
 datafile 'D:\EASYDW\dimensions.f' size 5m
 reuse autoextend on default storage
 (initial 16k next 16k pctincrease 0 maxextents unlimited);

-- Tablespace for the INDEXES
CREATE TABLESPACE easy_idx
 datafile 'D:\EASYDW\index.f'
 size 5m reuse autoextend on default storage
 (initial 16k next 16k pctincrease 0 maxextents unlimited);

-- Default Tablespace
CREATE TABLESPACE easydw_default
 datafile 'D:\EASYDW\easydw_default.f'
 size 5m reuse autoextend on default storage
 (initial 16k next 16k pctincrease 0 maxextents unlimited);

Once the tablespaces have been created for the dimensions, we can create the tablespaces for the fact table, "purchases." Since we will be partitioning the data, we must now create the tablespace for each partition. There will be one partition per month for the data and another partition for the indexes. Here we will create only the January partition for the data and index; simply repeat this process for the other partitions.

-- create the 3 month tablespaces for the fact partitions
CREATE TABLESPACE purchases_jan2002
 datafile 'D:\EASYDW\PURCHASESJAN2002.f' size 5m
 reuse autoextend on default storage
 (initial 16k next 16k pctincrease 0 maxextents unlimited);

-- create the 3 month tablespaces for the fact indexes
CREATE TABLESPACE purchases_jan2002_idx
 datafile 'D:\EASYDW\PURCHASESJAN2002_IDX.f' size 3M
 reuse autoextend on default storage
 (initial 16k next 16k pctincrease 0 maxextents unlimited);

2.5.2 Creating the tables, constraints, and indexes

Once the tablespaces have been defined, the EASYDW user can be created, which will create the schema where the data will be stored:

-- create a user called EASYDW
-- this will be the schema where the objects will reside

connect system/manager;

CREATE USER easydw identified by easydw
 default tablespace easydw_default
 temporary tablespace temp
 profile default account unlock;

GRANT unlimited tablespace to easydw ;
GRANT dba to easydw ;
GRANT create session TO easydw;

The DBA privilege has been granted to the users so they can create and manage the tables and indexes.

Hint:

Don't forget to connect as user EASYDW before creating the tables and indexes, or the tables and indexes will be defined in the wrong schema.

-- now create the database
CONNECT easydw/easydw;
-- CUSTOMER Dimension
CREATE TABLE easydw.customer
(customer_id                  varchar2(10),
 town                         varchar2(10),
 county                       varchar2(10),
 postal_code                  varchar2(10),
 dob                          date,
 country                      varchar2(20),
 occupation                   varchar2(10))
pctfree 0 pctused 99 tablespace easy_dim
storage (initial 16k next 16k pctincrease 0) ;

CREATE UNIQUE INDEX easydw.customer_pk_index ON customer
 (customer_id)
 pctfree 5
 tablespace easy_idx
 storage (initial 16k next 16k pctincrease 0) ;

ALTER TABLE customer
 ADD CONSTRAINT pk_customer PRIMARY KEY (customer_id);

We have defined the constraint here by adding it later. It could also be created within the CREATE TABLE definition, but if this approach is used, you cannot control the name of the index it creates by default to ensure fast validation of the primary key. Therefore, we create the index first, and, when the constraint is defined, it will use this index rather than create its own.

-- PRODUCT Dimension
CREATE TABLE easydw.product
(product_id                               varchar2(8),
 product_name                             varchar2(30),
 category                                 varchar2(4),
 cost_price                               number (6,2)
  constraint cost_price_not_null NOT NULL,
 sell_price                               number (6,2)
  constraint sell_price_not_null NOT NULL,
 weight                                   number (6,2),
 shipping_charge                          number (5,2)
  constraint shipping_charge_not_null NOT NULL,
 manufacturer                             varchar2(20),
 supplier                                 varchar2(10))
 pctfree 0 pctused 99
tablespace easy_dim
storage (initial 16k next 16k pctincrease 0) ;

CREATE UNIQUE INDEX easydw.product_pk_index ON product
 (product_id)
 pctfree 5 tablespace easy_idx
 storage (initial 16k next 16k pctincrease 0) ;
ALTER TABLE product
 ADD CONSTRAINT pk_product PRIMARY KEY (product_id);

Next the Time table is created. In this table, we have included a CHECK constraint to ensure that the column "public_holiday" can take only the values "Y" or "N."

-- TIME Dimension
CREATE TABLE easydw.time
(time_key                           date,
 day                                number (2,0),
 month                              number (2,0),
 quarter                            number (2,0),
 year                               number (4,0),
 day_number                         number (3,0),
 day_of_the_week                    varchar2(8),
 week_number                        number (2,0),
 public_holiday                     varchar2(1)
  constraint public_holiday
  CHECK (public_holiday IN ('Y','N')))
pctfree 0 pctused 99
tablespace easy_dim
storage (initial 16k next 16k pctincrease 0) ;

CREATE UNIQUE INDEX easydw.time_pk_index ON time
 (time_key)
 pctfree 5 tablespace easy_idx
 storage (initial 16k next 16k pctincrease 0) ;

ALTER TABLE time
 ADD CONSTRAINT pk_time PRIMARY KEY (time_key);

For the TODAYS_SPECIAL_OFFERS table, we have defined the primary key to include two columns rather than a single column.

-- TODAYS_SPECIAL_OFFERS Dimension
CREATE TABLE easydw.todays_special_offers
(product_id                         varchar2(8),
 offer_date                         date,
 special_price                      number (6,2),
 offer_price                        number (6,2))
pctfree 0 pctused 99
tablespace easy_dim
storage (initial 16k next 16k pctincrease 0) ;

CREATE UNIQUE INDEX easydw.tso_pk_index ON todays_special_offers
 (offer_date, product_id)
 pctfree 5 tablespace easy_idx
 storage (initial 16k next 16k pctincrease 0) ;
ALTER TABLE todays_special_offers
 ADD CONSTRAINT pk_specials PRIMARY KEY
          (offer_date,product_id);

Now we come to creating the all-important fact table, which is called PURCHASES. This table definition is quite complex, because it includes FOREIGN KEYS to several tables that are identified by the REFERENCES clause. For the column product_id, two constraints have been defined on the table: a NOT NULL and a foreign-key constraint. Provided each constraint is given a unique name, then it is allowed in a column in a table.

Here we have illustrated how to partition the table, which will be discussed in detail in Chapter 3.

-- Fact Table PURCHASES
CREATE TABLE easydw.purchases
(product_id                                      varchar2(8)
 constraint not_null_product_id NOT NULL
 constraint fk_product_id
       REFERENCES product(product_id),
 time_key                                        date
 constraint not_null_time NOT NULL
 constraint fk_time
       REFERENCES time(time_key),
 customer_id                                     varchar2(10)
 constraint not_null_customer_id NOT NULL
 constraint fk_customer_id
       REFERENCES customer(customer_id),
 purchase_date                                   date,
 purchase_time                                   number(4,0),
 purchase_price                                  number(6,2),
 shipping_charge                                 number(5,2),
 today_special_offer                             varchar2(1)
  constraint special_offer
  CHECK (today_special_offer IN ('Y','N')))
PARTITION by RANGE (time_key)
 (
      partition purchases_jan2002
       values less than (TO_DATE('01-02-2002', 'DD-MM-YYYY'))
       pctfree 0 pctused 99
       storage (initial 64k next 16k pctincrease 0)
       tablespace purchases_jan2002 ,
      partition purchases_feb2002
       values less than (TO_DATE('01-03-2002', 'DD-MM-YYYY'))
       pctfree 0 pctused 99
       storage (initial 64k next 16k pctincrease 0)
       tablespace purchases_feb2002 ,
      partition purchases_mar2002
       values less than (TO_DATE('01-04-2002', 'DD-MM-YYYY'))
       pctfree 0 pctused 99
       storage (initial 64k next 16k pctincrease 0)
       tablespace purchases_mar2002);

In this example, we have created the indexes immediately after the table definition. In a real data warehouse, the number of indexes created prior to loading the data is kept to an absolute minimum to ensure that the loading time is as fast as possible. Therefore, indexes exist usually only to check constraints.

-- Now create the indexes
-- Partition on the Time Key Local prefixed index
CREATE BITMAP INDEX easydw.purchase_time_index
 ON purchases (time_key)
local
 (partition indexJan2002 tablespace purchases_jan2002_idx,
 partition indexFeb2002 tablespace purchases_feb2002_idx,
 partition indexMar2002 tablespace purchases_mar2002_idx);

CREATE BITMAP INDEX easydw.purchase_product_index
  ON purchases (product_id)
 local
 pctfree 5 tablespace indx
 storage (initial 64k next 64k pctincrease 0) ;

CREATE INDEX easydw.purchase_customer_index
  ON purchases (customer_id)
 local
 pctfree 5 tablespace indx
 storage (initial 64k next 64k pctincrease 0) ;

CREATE BITMAP INDEX easydw.purchase_special_index
  ON purchases (today_special_offer)
 local
 pctfree 5 tablespace indx
 storage (initial 64k next 64k pctincrease 0) ;

2.5.3 Defining security

The next step is to grant some privileges to our user, EASYDW. We will start with the following ones, which will allow us to use Summary Management, and, as we progress through this book, we will discuss other privileges that should be granted to users.

connect system/manager;

-- Add privileges
GRANT SELECT ANY TABLE TO easydw;
GRANT EXECUTE ANY PROCEDURE TO easydw;

-- Add privileges for summary management
GRANT CREATE ANY DIMENSION TO easydw;
GRANT ALTER ANY DIMENSION TO easydw;
GRANT DROP  ANY DIMENSION TO easydw;
GRANT CREATE ANY MATERIALIZED VIEW TO easydw;
GRANT ALTER ANY MATERIALIZED VIEW TO easydw;
GRANT DROP  ANY MATERIALIZED VIEW TO easydw;
GRANT QUERY REWRITE TO easydw;
GRANT GLOBAL QUERY REWRITE TO easydw;

You will have to repeat these steps for every user, and the privileges granted will, of course, vary by user.

2.5.4 Final steps

The final step is actually not completed now but after the data is loaded. However, it is included here to remind you not to forget this important step, which is to analyze the table and indexes. These statistics are gathered using the package DBMS_STATS. The statistics gathered by the package are used by the optimizer. Without these statistics, features such as Summary Management will be unavailable. The DBMS_STATS command will be explained further in Chapter 3.

-- Now Analyze the Tables and Indexes

EXECUTE dbms_stats.gather_table_stats('EASYDW','CUSTOMER');
EXECUTE dbms_stats.gather_table_stats('EASYDW','TODAYS_SPECIAL_
OFFERS');
EXECUTE dbms_stats.gather_table_stats('EASYDW','product');
EXECUTE dbms_stats.gather_index_stats('EASYDW', 'purchase_
customer_index');

Our database is now complete. We have a basic framework, and in the next chapter we will learn how to enhance our basic database design to include and use the sophisticated features that are available in Oracle 9i.


Chapter 3: Data Warehouse Features

3.1 Optimizations for data warehousing

A data warehouse is primarily used to organize data so that queries for analysis and decision support can be answered quickly. As data warehouses grow in size, it is not uncommon to find a fact table several gigabytes or even terabytes in size. In order to obtain fast query response, it is extremely important for the database to retrieve and process such large amounts of data efficiently. Performance optimizations for update-centric OLTP systems are well understood; however, data warehouses, being query-centric, have vastly different requirements. In this chapter, we'll look at several features in the Oracle database specifically designed to improve query performance in a data warehouse.

Good performance in a warehouse is largely dependent on sound physical database design. We will look at several techniques, including partitioning, data segment compression, and bitmap indexing, that are suited for data warehouses. We will also discuss query optimizations, such as partition pruning, partition-wise join, and star transformation.

Decision-support applications often need to perform computations such as period-over-period comparisons and cumulative aggregations. These queries traditionally required extremely complex and lengthy SQL. Oracle and several other database vendors have started an effort to provide standardized extensions to the SQL language to allow users to express such queries simply and process them efficiently. We will look at some of these new SQL functions.


Indexing

3.2 Indexing

Indexing has always been a very important technique for efficient query processing in database systems. Unlike OLTP systems, which have mostly update activity, data warehouses tend to read large amounts of data to answer queries. Hence, it is important to understand the indexing needs of a warehouse.

Deciding which indexes to create is an important part of the physical design of a database. Indexes should be built on columns that are often part of the selection criteria of a query. Columns that are frequently referenced in the SQL WHERE clause are good candidates for indexing. Most decision-support queries require specific rows from a dimension table, and so it is important to have good indexing on the dimension tables. For example, if a user is looking for the location of all stores in the Northeast, as in the following query, an index built on the column "region" could be used to locate just those rows in the Northeast rather than reading every row in the table.

SELECT store, location FROM stores WHERE region = 'Northeast';

Typical warehouses have low update activity, other than for loading new data. Hence, a warehouse can have many more indexes than an OLTP system. Space requirements for indexes in a warehouse are often significantly larger than the space needed to store the data, especially for the fact table. Therefore, you may want to keep indexing on the fact table to a minimum. Typically, you may have one or two concatenated indexes on the fact table. You may also want to consider bitmap indexes on foreign-key columns to speed up star joins. On the other hand, dimension tables are much smaller compared with the fact table and could be indexed much more extensively. Any column of the dimension table that is frequently used in selections or is a level in a dimension object, described in Chapter 4, is a good candidate for indexing.

An index is built on one or more columns of a table. These columns are known as the index keys. For each key value, the index contains a pointer to the location of the rows with that key value. Whenever data in the table changes, the index is automatically updated to reflect the changed data.

Oracle offers several kinds of indexes. We will discuss three of these in this book:

  • B*tree index

  • Bitmap index

  • Bitmap join index

3.2.1 B*tree indexes

B*tree indexes are hierarchical structures that allow rapid searching in order to obtain the address of a row in a table with a particular value. B*tree indexes store pointers to rows of the tables using rowids. There are two varieties of b*tree indexes:

  • Unique

  • Non-unique

A unique index ensures that each row has a distinct value for its key. No duplicates are allowed. A unique index is automatically created when a PRIMARY KEY or UNIQUE constraint is enabled on a table. B*tree indexes can also be nonunique. Nonunique indexes improve query performance when a small number of rows are selected.

When multiple columns are used together in the WHERE clause, you can build an index on that group of columns. Indexes made up of multiple columns are called composite, multikey, or concatenated indexes. For example, city and state are both needed to differentiate Portland, Maine, from Portland, Oregon. The column that is used most frequently by itself should be specified as the first column in the index. In the example, state should be the leading column, since we can anticipate queries on state alone or city and state used together.

Whenever a row is inserted, deleted, or when the key columns are updated, the index is automatically updated to reflect the change. A b*tree index is designed such that the time required to search any particular value in the index is nearly constant. This design is called a balanced index. As new index nodes are created, the b*tree is automatically rebalanced. Index maintenance, therefore, adds overhead to the DML statement. When loading a large number of rows, it is sometimes faster to mark the index unusable for the duration of the insert and rebuild the index, as shown in the following code segment:

ALTER INDEX customer_gender_index UNUSABLE;

INSERT /*+ APPEND */ INTO customers ...
COMMIT;

ALTER INDEX customer_gender_index REBUILD;

A b*tree index is useful when the index key has many distinct values, each leading to a few rows in the table. For instance, a primary key implies each value of the key corresponds to one row in a table. However, if the index-key column has only a few distinct values, then each value would lead to retrieval of large numbers of rows in the table. This provides little, if any, performance benefit. For instance, consider the query: How many women who live in California buy tents? There are only two possible values for the column "sex"; hence, finding all the rows corresponding to women may result in reading all the data blocks in the table, since multiple records are typically stored in one block. A full table scan may be more efficient. Also, for large tables, space requirements for b*tree indexes could become prohibitive.

Bitmap indexes are designed to solve these problems and, hence, are more commonly used in a warehouse.

3.2.2 Bitmap indexes

Bitmap indexes are designed to answer queries involving columns with few distinct values but potentially large numbers of rows for each value. The number of distinct values of a column is known as its cardinality. Unlike b*tree indexes, which store pointers to rows of the tables using rowids, a bitmap index stores a bitmap for each distinct value of a column.

The bitmap for each distinct value has as many bits as the number of rows in the table. The bit is set if the corresponding row has that value. Table 3.1 shows a bitmap index for the "sex" column (cardinality = 2, distinct values - M and F). Two bitmaps are created: one for the value M and one for the value F. In the bitmap for M, all rows for male customers would have their bits set to 1 and all rows for female customers would have their bits set to 0.

Table 3.1: Sex Bitmap Index
Row | Sex | Male Bitmap | Female Bitmap
1 | M | 1 | 0
2 | F | 0 | 1
3 | M | 1 | 0
4 | M | 1 | 0
5 | M | 1 | 0
6 | F | 0 | 1
7 | M | 1 | 0
8 | M | 1 | 0
9 | F | 0 | 1
10 | M | 1 | 0

Bitmap indexes on two columns of a table can be combined efficiently using the AND and OR Boolean operators. This allows them to be used for queries involving multiple conditions on bitmapped columns of a table. This gives the benefit that you can answer a wide variety of queries with just a few single-column bitmap indexes. For instance, suppose we had a customer table that contained the customer name, sex, and occupation (teacher, engineer, self-employed, housewife, doctor, etc.). We are looking at a new promotion targeted toward self-employed women and would like to find out the towns to target. This can be expressed as a query:

SELECT customer_name, town
FROM customer
WHERE sex = 'F' AND occupation = 'Self-Employed';

Table 3.2 shows the bitmap for the occupation column.

Table 3.2: Occupation Bitmap Index
Row | Occupation | Teacher | Engineer | Self-Employed | Housewife
1 | Teacher | 1 | 0 | 0 | 0
2 | Engineer | 0 | 1 | 0 | 0
3 | Teacher | 1 | 0 | 0 | 0
4 | Teacher | 1 | 0 | 0 | 0
5 | Engineer | 0 | 1 | 0 | 0
6 | Self-Employed | 0 | 0 | 1 | 0
7 | Self-Employed | 0 | 0 | 1 | 0
8 | Housewife | 0 | 0 | 0 | 1
9 | Engineer | 0 | 1 | 0 | 0
10 | Housewife | 0 | 0 | 0 | 1

The sex bitmap defined in Table 3.1 and the occupation bitmap defined in Table 3.2 can be combined together, as shown in Table 3.3, to quickly determine the rows in the customer table that satisfy this query.

Table 3.3: Using a Bitmap Index—How many customers are self-employed women?
Row | Female Bitmap | Self-Employed | Result
1 | 0 | 0 | 0
2 | 1 | 0 | 0
3 | 0 | 0 | 0
4 | 0 | 0 | 0
5 | 0 | 0 | 0
6 | 1 | 1 | 1 (self-employed women)
7 | 0 | 1 | 0
8 | 0 | 0 | 0
9 | 1 | 0 | 0
10 | 0 | 0 | 0

The SQL statement used to create the bitmap index on the customer.sex column is as follows:

CREATE BITMAP INDEX easydw.customer_gender_index
 ON customer(sex)
pctfree 5 tablespace indx
storage (initial 64k next 64k pctincrease 0);

Alternatively, you can create a bitmap index in Oracle Enterprise Manager, as shown in Figure 3.1. In order to create a bitmap index rather than a b*tree index, check the bitmap box. You can specify storage parameters and see the SQL generated for the index creation.

Click To expand
Figure 3.1: Creating a bitmap index in Oracle Enterprise Manager.

Bitmap indexes can be compressed and therefore require much less space than a corresponding b*tree index. The space savings can be significant if the number of distinct values of the bitmap is small compared with the number of rows in the table.

The disadvantage of bitmap indexes is that they are expensive to maintain when the data in the table changes. This is because the bitmaps must be uncompressed, recompressed, and possibly rebuilt. Fortunately, in a warehouse individual rows are updated very rarely. Bitmap index maintenance is optimized for the common case of loading new data into a warehouse via bulk insert. In this case, existing bitmaps do not need change and are simply extended to include the new rows. Also, in the case of a b*tree index, when updating a row, only that row is locked; however, with bitmap indexes, a large part of the bitmap may need to be locked. Thus, bitmap indexes reduce the concurrency in the system. Hence, bitmap indexes are not suited for systems with a lot of concurrent update activity.

3.2.3 Bitmap join index

Oracle 9i introduced a variation on bitmap indexes called the bitmap join index. A bitmap join index can be used to index the result of a join between a fact table and one or more dimension tables. To illustrate the concept of bitmap join indexes, consider the following query: How much did women spend in our store?:

SELECT sum(p.purchase_price)
FROM purchases p, customer c
WHERE p.customer_id = c.customer_id
   AND c.sex = 'F'

We could create a bitmap index on the customer.sex column, as discussed in the previous section to find those customers who are women quickly. However, we would still need to compute the join between purchases and customers to find the purchases made by women.

Instead, if we take the answer to the join and then create a bitmap to identify rows corresponding to purchases made by men and women, we get a bitmap join index. Table 3.4 shows the answer to the previous join between purchases and customer table (only the first ten rows are shown).

Table 3.4: Join On Purchases and Customer Tables
Row | Customer_id | Sex | Purchase_price
1 | AB123456 | F | 28.01
2 | AB123457 | F | 28.01
3 | AB123457 | F | 28.01
4 | AB123457 | F | 28.01
5 | AB123456 | F | 67.23
6 | AB123457 | F | 67.23
7 | AB123458 | M | 67.23
8 | AB123459 | M | 67.23
9 | AB123460 | F | 67.23
10 | AB123461 | M | 50.71
... |  |  |

The bitmap join index has two bitmaps one for the value male and another for the value female, as shown in Table 3.5. Each row in the bitmap corresponds to a single row in the purchases table. Thus, we can immediately identify the rows for purchases made by women from the bitmap for value female.

Table 3.5: Bitmap Join Index on Purchases and Customer on customer.sex Column
Row | Sex | Male Bitmap | Female Bitmap
1 | F | 0 | 1
2 | F | 0 | 1
3 | F | 0 | 1
4 | F | 0 | 1
5 | F | 0 | 1
6 | F | 0 | 1
7 | M | 1 | 0
8 | M | 1 | 0
9 | F | 0 | 1
10 | M | 1 | 0
... |  |  |

The bitmap join index on purchases, joined with customer, is created using the following SQL statement. The joining tables and their join conditions are specified using the FROM and WHERE clauses, similar to those in a SELECT statement. The table on which the index is built is specified by the ON clause, as with any other index.

CREATE BITMAP INDEX easydw.purchase_cust_index
ON
purchases (customer.sex)
FROM purchases, customer
WHERE purchases.customer_id = customer.customer_id;

We can immediately note some differences between a bitmap index and a bitmap join index. While a bitmap index is created on columns of a single table, a bitmap join index is built on the fact table (purchases), but the index columns are from dimension tables (customer).

As with a bitmap index, columns included in a bitmap join index must be low-cardinality columns. For a bitmap join index to make sense, the result of the join should have the same number of rows as the fact table on which it is created (in our example the purchases table). To ensure this, a unique constraint must be present on the dimension table column that joins it to the fact table (in our example the customer.customer_id column).

Bitmap join indexes put some restrictions on concurrent DML activity on the tables involved in the index, but this is not a major issue for most warehouse applications.

An alternative to a bitmap join index is a materialized view with joins. We will briefly compare the two in Chapter 4.


Data partitioning

3.3 Data partitioning

3.3.1 Benefits of partitioning

Whenever any task seems daunting, breaking it up into smaller tasks often makes it easier to accomplish. Imagine packing your belongings, and getting ready to move. Dividing it up room by room makes it easier. If each member of the family packs up a room at the same time, you can get all your belongings packed faster. This is the idea behind partitioning and parallelism.

Very large tables and indexes can be divided into smaller, more manageable partitions. Partitioning the data makes it possible for data management operations to be performed at the partition level. Many operations, such as loading data, building local indexes, enforcing constraints, gathering optimizer statistics, purging data, and backup and recovery, can be done at the granularity of a partition.

Partitioning of data also allows queries on the tables to take advantage of the parallel processing features of Oracle. In addition, DML operations on the table can be done in parallel, providing significant improvements in performance.

If a table is partitioned, the query optimizer can determine if a certain query can be answered by reading only specific partitions. Thus, an expensive table scan can be avoided. This feature is known as partition elimination or dynamic partition pruning. If the warehouse was partitioned by month and a query asked for sales from December 1998, the optimizer would know which partition the data was stored in, and would eliminate partitions containing other months from its search.

3.3.2 How to partition data

A table can be partitioned using any column of the table. This column is known as the partition key. It is important to partition the data by a column that does not change, such as the purchase date. If partitioning is done by product_id and the business changed the encoding scheme for its products, then updating the partitioning column would require moving data to a different partition, which can be a time-consuming operation.

In data warehouses, it is common to partition by time. This allows us to perform "rolling window operations." If our warehouse contained one year's worth of data, at the end of April 2002 we could add a new partition with that month's data and delete the data for April 2001. In Chapter 5 we will look at loading data into a new partition.

Partitioning improves the availability of the data warehouse. By placing each partition on its own disk, if one disk fails and is no longer accessible, only the data in that partition is unavailable-not the entire table. Maintenance operations can be performed on one partition, while users continue to access data from the other partitions in the table.

For maximum performance, each partition should be stored in a separate tablespace and each tablespace should be stored on one or more separate storage devices, as appropriate. On larger systems, the I/O should also be spread across different controllers and channels. By spreading the data across several physical devices, you balance the I/O and improve query performance, availability, and manageability.

There are several ways to partition your data, including the following:

  • By ranges of data values (range partitioning)

  • By a hash function (hash partitioning)

  • Using discrete values (list partitioning)

  • Using a combination of methods (composite partitioning)

Range partitioning

In range partitioning, data is partitioned into non-overlapping ranges of values. As each row gets inserted into the table, it is placed into the appropriate partition based on the value of the partition-key column. Range partitioning is especially useful when the partition key is time. It allows the optimizer to perform partition pruning for queries asking for a specific value or a range of partition-key values.

In the EASYDW schema, each month's purchases are stored in their own partition. The purchases fact table is partitioned by range, using the time_ key as the partition key. Each partition is stored in its own tablespace, spread across multiple disks, as illustrated in Figure 3.2.

Click To expand
Figure 3.2: Range partitioning.

The SQL that would create this partitioned table is shown in the following code segment. The VALUES LESS THAN clause specifies the upper bound on the partition-key values in that partition. The lower bound is specified by the VALUES LESS THAN clause of the previous partition, if any. In our example, the purchases_feb2002 partition has data values with time_key ≥01-02-2002 and <01-03-2002.

CREATE TABLE easydw.purchases
(product_id varchar2(8),
 time_key date,
 customer_id varchar2(10),
 purchase_date date,
 purchase_time number(4,0),
 purchase_price number(6,2),
 shipping_charge number(5,2),
 today_special_offer varchar2(1))
PARTITION by RANGE (time_key)
 (partition purchases_jan2002
    values less than (TO_DATE('01-02-2002', 'DD-MM-YYYY'))
    pctfree 0 pctused 99
    storage (initial 64k next 64k pctincrease 0)
    tablespace purchases_jan2002,
  partition purchases_feb2002
    values less than (TO_DATE('01-03-2002', 'DD-MM-YYYY'))
    pctfree 0 pctused 99
    storage (initial 64k next 64k pctincrease 0)
    tablespace purchases_feb2002,
  partition purchases_mar2002
    values less than (TO_DATE('01-04-2002', 'DD-MM-YYYY'))
    pctfree 0 pctused 99
    storage (initial 64k next 64k pctincrease 0)
    tablespace purchases_mar2002,
  partition purchase_catchall
    values less than (MAXVALUE)
    pctfree 0 pctused 99
    storage (initial 64k next 64k pctincrease     0)
    tablespace purchases_maxvalue);

Note that the last partition in the purchases table has a special bound called MAXVALUE. This is an optional catchall partition, which collects all rows that do not correspond to any defined partition ranges.

Hash partitioning

With range partitioning, it is possible to end up with a situation where the data is not evenly divided among the partitions. Some partitions may be very large and others small. If the data was partitioned by month, and some months had more sales than others, this would result in partitions that are very different in size. When the data is skewed in this way, hot spots form where there is contention for resources in one area.

Hash partitioning reduces this type of data skew by applying a hashing function to the partitioning key. The resulting output value is used to determine which partition to store the row in. So instead of partitioning by month, if we hash partitioned the purchases table by product_id, all rows for a month would get scattered across several partitions, as shown in Figure 3.3. Notice that all rows for the same product fall into the same partition; however, there is no way to specify which partition will have which products.

Click To expand
Figure 3.3: Hash partitioning.

The following example illustrates the SQL required to create a hash partitioned table.

CREATE TABLE easydw.purchases
 (product_id          varchar2(8),
 time_key             date,
 customer_id          varchar2(10),
 purchase_date        date,
 purchase_time        number(4,0),
 purchase_price       number(6,2),
 shipping_charge      number(5,2),
 today_special_offer  varchar2(1))
PARTITION BY HASH(product_id)
PARTITIONS 3;

Hash partitioning is good for finding rows corresponding to a single value but cannot be used by the optimizer to perform partition elimination for queries requesting a range of values. If we had a query based on a specific product, such as: How many tents did we sell each month? the optimizer could determine which partition to examine. However, if we had a query: What products did we sell during the month of January? we would need to search all partitions.

In hash partitioning, the user has no control or knowledge of which products go into which partitions. You can only specify how many partitions you would like. This makes it difficult to do maintenance operations that require knowledge of the data, such as archiving old data from the table. Hence, hash partitioning is not commonly used by itself but is used in conjunction with range partitioning.

List partitioning

In some cases, it may not be convenient to organize data into ranges of values. The data may not have a natural partitioning key such as time. For business reasons, values that are far apart may need to be grouped together. For instance, if we have sales data for various states of the United States, it is not very easy to put data for all states in New England into the same partition. In Oracle 9i, a new type of partitioning was introduced to solve this problem. This partitioning method, known as list partitioning, allows data to be distributed according to discrete column values. Figure 3.4 shows an example of a list-partitioned table.

Click To expand
Figure 3.4: List partitioning.

The following SQL statement uses list partitioning to organize the sales data by region. The last partition is a catchall partition, which captures the rows that do not map to any other partition.

CREATE TABLE easydw.regional_sales
(state            varchar2(2),
 store_number     number,
 dept_number      number,
 dept_name        varchar2(10),
 sales_amount     number (6,2)
)
PARTITION BY LIST(state)
(
 PARTITION northeast VALUES ('NH', 'VT', 'MA', 'RI', 'CT'),
 PARTITION southeast VALUES ('NC', 'GA', 'FL'),
 PARTITION northwest VALUES ('WA', 'OR'),
 PARTITION midwest  VALUES ('IL', 'WI', 'OH'),
 PARTITION west   VALUES ('CA', 'NV', 'AZ'),
 PARTITION otherstates VALUES (DEFAULT));

List partitioning allows the query optimizer to perform partition pruning on queries that ask for specific values of the partitioning key. For instance, a query requesting data for Massachusetts or New Hampshire only needs to access the northeast partition.

Composite partitioning

Oracle provides a two-level partitioning scheme known as composite partitioning to combine the benefits of two partitioning methods. In composite partitioning, data is divided into partitions using one partitioning scheme and further subdivided into subpartitions using another scheme. Currently, Oracle supports two types of composite partitioning schemes:

  • Range-Hash

  • Range-List

We mentioned earlier that hash partitioning does not allow a user control over the distribution of data. On the other hand, range partitioning suffers from the potential problem of data skew. Range-hash composite partitioning combines the benefits of range and hash partitioning. The data is first partitioned by range and then further subdivided into subpartitions by using a hash function. When partitioning by date, the last partition is often a hot spot. By further subpartitioning using a hash function, I/O contention is reduced because access is divided among the subpartitions.

All partitions of a table or index have the same logical attributes but can have different physical attributes. For example, all partitions in a table have the same column and constraint definitions, but they can have different storage attributes and physical placement. Subpartitions of a single partition can be placed in different tablespaces. This allows data to be distributed across many physical devices to reduce I/O contention.

In Figure 3.5, the data is first partitioned by month and then further partitioned by product. Each partition has three subpartitions. Each subpartition has been stored in its own tablespace. The SQL to create a composite partitioned table is shown in the following example. The STORE IN clause allows you to name the tablespace where each subpartition will reside.

Click To expand
Figure 3.5: Range-hash composite partitioning.
CREATE TABLE easydw.purchases
(product_id varchar2(8),
 time_key date,
 customer_id varchar2(10),
 purchase_date date,
 purchase_time number(4,0),
 purchase_price number(6,2),
 shipping_charge number(5,2),
 today_special_offer varchar2(1))
PARTITION by RANGE (time_key)
  SUBPARTITION BY HASH(product_id)
  SUBPARTITIONS 3
 (partition purchases_jan2002
   values less than (TO_DATE('01-02-2002', 'DD-MM-YYYY'))
   pctfree 0 pctused 99
   storage (initial 64k next 16k pctincrease 0)
   STORE IN (purchases_jan2002,
             purchases_jan2002_2,
             purchases_jan2002_3),
  partition purchases_feb2002
   values less than (TO_DATE('01-03-2002', 'DD-MM-YYYY'))
   pctfree 0 pctused 99
   storage (initial 64k next 16k pctincrease 0)
   STORE IN (purchases_feb2002,
             purchases_feb2002_2,
             purchases_feb2002_3),
  partition purchases_mar2002
   values less than (TO_DATE('01-04-2002', 'DD-MM-YYYY'))
   pctfree 0 pctused 99
   storage (initial 64k next 16k pctincrease 0)
   STORE IN (purchases_mar2002,
             purchases_mar2002_2,
             purchases_mar2002_3));

The range-list composite partitioning method first partitions a table by a continuous key, such as time_key, and then subpartitions each partition with discrete values. In the following example, we have a table range partitioned by month and further list partitioned by state. A query that asks for sales for the dates in the month of January for the New England states can be evaluated efficiently if the sales table is partitioned using range-list partitioning.

CREATE TABLE sales
(state varchar2(2),
 store_number number,
 dept_number number,
 dept_name varchar2(10),
 sales_amount number (6,2),
 sale_date date,
 item_number number (10)
)
PARTITION BY RANGE (sale_date)
  SUBPARTITION BY LIST(state)
  SUBPARTITION TEMPLATE
  (SUBPARTITION "NorthEast" VALUES ('NH','VT','MA','RI','CT')
                              TABLESPACE sales_ne,
   SUBPARTITION "SouthEast" VALUES ('NC','GA','FL')
                              TABLESPACE sales_se,
   SUBPARTITION "NorthWest" VALUES ('WA','OR')
                              TABLESPACE sales_nw,
   SUBPARTITION "MidWest"   VALUES ('IL','WI','OH')
                              TABLESPACE sales_mw,
   SUBPARTITION "West"      VALUES ('CA','NV','AZ')
                              TABLESPACE sales_w)
(
  PARTITION sales_jan_2002
   VALUES LESS THAN (TO_DATE('01-Feb-2002', 'DD-Mon-YYYY')),
  PARTITION sales_feb_2002
   VALUES LESS THAN (TO_DATE('01-Mar-2002', 'DD-Mon-YYYY')),
  PARTITION sales_mar_2002
   VALUES LESS THAN (TO_DATE('01-Apr-2002', 'DD-Mon-YYYY'))
);

In this example, we show the use of the SUBPARTITION TEMPLATE clause. The specification of range-list partitioning can get quite verbose, since you have to specify the detailed subpartition clause for each range partition. Often, you want the same list subpartitioning within each of your range partitions. The SUBPARTITION TEMPLATE makes it convenient to specify the same subpartition information for all range partitions in the table. Oracle will generate the subpartition name using a combination of the partition name and the name specified in the template. It then generates the subpartitions according to the definition in the template. The partition and subpartition information for a table can be obtained from the user_tab_partitions and user_tab_subpartitions dictionary views. For our example, the partition and subpartition names are as follows:

SELECT partition_name, subpartition_name
FROM user_tab_subpartitions
WHERE table_name = 'SALES';

PARTITION_NAME         SUBPARTITION_NAME
---------------------- ------------------------------
SALES_JAN_2002         SALES_JAN_2002_NorthEast
SALES_JAN_2002         SALES_JAN_2002_SouthEast
SALES_JAN_2002         SALES_JAN_2002_NorthWest
SALES_JAN_2002         SALES_JAN_2002_MidWest
SALES_JAN_2002         SALES_JAN_2002_West
SALES_FEB_2002         SALES_FEB_2002_NorthEast
SALES_FEB_2002         SALES_FEB_2002_SouthEast
SALES_FEB_2002         SALES_FEB_2002_NorthWest
SALES_FEB_2002         SALES_FEB_2002_MidWest
SALES_FEB_2002         SALES_FEB_2002_West
...

You can choose not to use the SUBPARTITION TEMPLATE but instead specify the list sub-partition values and names for each range partition.

The following query asks for total sales for NH, MA and CT states for a range of dates. The optimizer will determine that this query can be answered quickly by reading the first sub-partition of the partitions for February and March. The Pstart and Pstop values in the output of EXPLAIN LAN below shows the range of partitions used to answer the query, in this case partitions 2 to 3.

EXPLAIN PLAN FOR
SELECT store_number, dept_number, SUM(sales_amount) as q1_sales
FROM sales
WHERE sale_date between TO_DATE('15-Feb-2002', 'DD-Mon-YYYY')
  and TO_DATE('15-Mar-2002', 'DD-Mon-YYYY')
  and state in ('NH', 'MA', 'CT')
GROUP BY store_number, dept_number;

-----------------------------------------------------------------
|Id| Operation                 |Name |Rows|Cost| Pstart | Pstop |
-----------------------------------------------------------------
|0 | SELECT STATEMENT          |     | 1  | 16 |        |       |
|1 |  SORT GROUP BY            |     | 1  | 16 |        |       |
|2 |   PARTITION RANGE ITERATOR|     |    |    |     2  |     3 |
|3 |    PARTITION LIST INLIST  |     |    |    |KEY(I)  |KEY(I) |
|4 |     TABLE ACCESS FULL     |SALES| 1  | 14 |   KEY  |   KEY |
-----------------------------------------------------------------
Hint:

Range-list partitioning is particularly suited as a partitioning scheme for materialized views, which typically contain summarized data across multiple dimensions, one of which is often time.

Multicolumn partition keys

A partition key used for range partitioning can have multiple columns. The following SQL code segment creates a table with range partitioning using a multicolumn partitioning key.

CREATE TABLE easydw.purchases
(product_id varchar2(8),
 time_key date,
 customer_id varchar2(10),
 purchase_date date,
 purchase_time number(4,0),
 purchase_price number(6,2),
 shipping_charge number(5,2),
 today_special_offer varchar2(1))
PARTITION by RANGE (time_key, product_id)
  (partition purchases_jan2002_100
     values less than (TO_DATE('31-01-2002','DD-MM-YYYY'), 100)
     pctfree 0 pctused 99
     storage (initial 64k next 64k pctincrease 0)
     tablespace purchases_jan2002_100,
   partition purchases_jan2002_200
     values less than (TO_DATE('31-01-2002','DD-MM-YYYY'), 200)
     pctfree 0 pctused 99
     storage (initial 64k next 64k pctincrease 0)
     tablespace purchases_jan2002_200 ,
   partition purchases_feb2002_all
     values less than (TO_DATE('28-02-2002','DD-MM-YYYY'), 100)
     pctfree 0 pctused 99
     storage (initial 64k next 64k pctincrease 0)
     tablespace purchases_feb2002,
   partition purchases_mar2002_all
     values less than (TO_DATE('31-03-2002','DD-MM-YYYY'), 100)
     pctfree 0 pctused 99
     storage (initial 64k next 64k pctincrease 0)
     tablespace purchases_mar2002
);

To understand how data get mapped to partitions, let us now insert some data values into this table and see which partitions they go into.

insert into purchases (product_id, time_key)
    values (1, TO_DATE('15-01-2002', 'DD-MM-YYYY'));
insert into purchases (product_id, time_key)
    values (150, TO_DATE('15-01-2002', 'DD-MM-YYYY'));
insert into purchases (product_id, time_key)
    values (101, TO_DATE('31-01-2002', 'DD-MM-YYYY'));
insert into purchases (product_id, time_key)
    values (170, TO_DATE('31-01-2002', 'DD-MM-YYYY'));
insert into purchases (product_id, time_key)
    values (200, TO_DATE('31-01-2002', 'DD-MM-YYYY'));
insert into purchases (product_id, time_key)
    values (1, TO_DATE('28-02-2002', 'DD-MM-YYYY'));

SELECT product_id, time_key
FROM purchases partition(purchases_jan2002_100);

PRODUCT_ID TIME_KEY
---------- -----------
1          15-JAN-2002
150        15-JAN-2002

SELECT product_id, time_key
FROM purchases partition(purchases_jan2002_200);

PRODUCT_ID TIME_KEY
---------- -----------
101        31-JAN-2002
170        31-JAN-2002

SELECT product_id, time_key
FROM purchases partition(purchases_feb2002_all);

PRODUCT_ID TIME_KEY
---------- -----------
200        31-JAN-2002
1          28-FEB-2002

Notice that the row with 15-Jan-2002 for product_id = 150 goes into the first partition rather than the second. This may not be what you expect. In this case, the condition checked for the first partition is (time_key < '31-01-2002') or (time_key = '31-01-2002' and product_id < 100). If the (time_key, product_id) values do not map to the first partition, the second partition is checked using (time_key < '31-01-2002') or (time_key = '31-01-2002' and product_id < 200). Multikey partitioning can be useful if you have one time_key value with lots of product_id values (e.g., several purchases may be made on Christmas Eve). In this case, you can partition on that value to distribute the data into several partitions.

A range-partitioning scheme with a multicolumn partition key must not be confused with range-range composite partitioning. As of the time of writing, Oracle does not support range-range composite partitioning. If it were supported, the row with time_key = 15-Jan-2002 and product_id = 150 would indeed map to the second partition in the previous example.

To summarize, range partitioning should be used when your table has a continuous key, such as time. List partitioning is ideal for tables where you would like to place specific discrete values in one partition. Hash partitioning distributes data uniformly among all partitions and can be used in combination with range partitioning to avoid hot spots in the table. Finally, range-list partitioning may be used when the table stores data along multiple dimensions: one continuous, such as time, and the other discrete, such as product or geography. Use of partitioning provides great benefits in a warehouse by improving query performance, manageability, and availability of data.

In the next section, we will see how partitioning can be applied to an index.

3.3.3 Partitioned indexes

Both b*tree and bitmap indexes can be partitioned using any of the available partitioning methods. They may be partitioned the same way that the table is partitioned or by different criteria. A b*tree index may be partitioned even if the underlying table is not. Partitioned indexes can be of two types:

  • Global

  • Local

Global indexes

In a global index, the keys in an index partition need not correspond to any single table partition or subpartition. A global index may not be partitioned at all or can have a partition key completely different from the table. However, a partitioned global index must have the partitioning key as the leading column of the index key. Bitmap indexes on partitioned tables cannot be global.

Figure 3.6 shows a global index purchases_product_index on the purchases table. Here, the purchases table is partitioned by time_key, but the index is partitioned by product_id. The leading column of the index is also product_id.

Click To expand
Figure 3.6: Global index.

The following code segment shows the SQL used to create this index:

CREATE INDEX easydw.global_product_index on purchases
 (product_id)
 global
 partition by range (product_id)
 (partition sp1000 values less than ('SP1000') ,
  partition sp2000 values less than ('SP2000') ,
  partition other values less than (maxvalue));

Figure 3.6 shows why global indexes are not very efficient for query processing. Since there is no correlation between the table and index partitions, accessing values in a table partition can involve access to several or all of the index partitions.

In addition, when the data in an underlying table partition is moved or removed using a partition maintenance operation, all partitions of a global index are affected and the index is marked UNUSABLE. An ALTER INDEX REBUILD statement must be issued to rebuild the index. (Note that in Oracle 9i release2, there is an option to efficiently rebuild the affected global indexes by specifying the UPDATE GLOBAL INDEXES clause on the partition operation.)

These shortcomings of a global index are addressed by local indexes.

Local indexes

A local index inherits its partitioning criteria from the underlying table. It has the same number of partitions, subpartitions, and partition bounds as the table. When partitions are added, dropped, split, or merged in the underlying table, the index partitions are automatically modified correspondingly.

There are two types of local partitioned indexes:

  • Prefixed

  • Non-prefixed

If the partitioning key of the index appears as a leading prefix of the index, it is called a prefixed index. Otherwise, the index is said to be non-prefixed.

Prefixed local indexes

In our example, the purchases table is partitioned by the time key. Figure 3.7 shows an example of a prefixed local index, purchase_time_index on this table. The partitioning key is time_key and the index columns are (time_key, customer_id). The partitioning key, time_key, is a prefix of the index key.

Click To expand
Figure 3.7: Prefixed local index.

Following is the SQL required to create this index. The local index follows the same partitioning scheme as the table, but you can name the partitions and also the tablespace where each partition resides.

CREATE INDEX easydw.purchase_time_index
 ON purchases (time_key, customer_id)
local
 (partition indexJan2002 tablespace purchases_jan2002_idx,
  partition indexFeb2002 tablespace purchases_feb2002_idx,
  partition indexMar2002 tablespace purchases_mar2002_idx);

With this index, if we wanted to know the purchases made by a certain customer in January, we need only to search the partition indexJan2002.

Nonprefixed local indexes

Local indexes that do not include the partitioning key as the leading column of the index are known as nonprefixed local indexes. Such indexes are useful when we would like to partition by one column for ease of maintenance but index other columns for data retrieval. For instance, we may want our indexes and tables to be partitioned by time_key, so it's easy to add a new month's data and build data for that month. However, to get good performance for queries for sales by product_id, we need an index on product_id.

Figure 3.8 shows a local nonprefixed index on the product_id column of the purchases table. The partitioning scheme is the same as that of the purchases table (i.e., on the time_key column). The following code segment shows how to create this index.

Click To expand
Figure 3.8: Nonprefixed local index.
CREATE BITMAP INDEX easydw.purchase_product_index
ON purchases  (product_id)
local
 (partition indexJan2002 tablespace purchases_jan2002_idx,
  partition indexFeb2002 tablespace purchases_feb2002_idx,
  partition indexMar2002 tablespace purchases_mar2002_idx);

With this index, when searching for sales of some product, say "Tents" the optimizer is not able to perform partition elimination and must search for the data for all months. However, when searching for "Tents" sold in January, it can eliminate the February and March partitions of the index.

To summarize, local prefixed indexes are the most efficient type of indexes for query performance, since the optimizer can use partition elimination to avoid looking at unnecessary partitions. Local indexes also facilitate rolling window operations. Avoid using global indexes if possible, since additional maintenance is required and bitmap indexes cannot be global.

3.3.4 RAID technology

An alternative technique for spreading the I/O requests across multiple devices is by using RAID storage technology-redundant arrays of inexpensive disks. By using multiple disks together, performance and reliability can be improved. Rather than storing all data in one file on a single disk, the data is striped across multiple disks, reducing contention for disk arms. RAID technologies can be used with all the data warehousing features discussed in this book.


Data compression

3.4 Data compression

Oracle performs data compression in a variety of ways to improve storage requirements of data for indexes and tables. The benefits of compression are greatest when the data has many repeated values (such as tables with foreign keys) and if data values are not updated frequently.

For b*tree indexes, Oracle performs key compression by storing the common prefix of index key values in an index block, only once. The number of columns to be used in the prefix is equal to the number of key columns - 1 for unique indexes and equal to the number of key columns for nonunique indexes. Index-key compression must be enabled using the COMPRESS keyword on the CREATE INDEX statement.

For bitmap indexes, the bitmap for each data value is automatically compressed by Oracle. For columns whose cardinality is much less than the number of rows in the table, the bitmaps tend to be dense (having a lot of 1s) and can be compressed greatly. Thus, a bitmap index can occupy much less space than the corresponding b*tree index.

Another very useful type of compression is data segment compression, introduced in Oracle 9i Release 2. Data segment compression identifies repeated values of data in a data block and places them in a lookup table at the beginning of the block. The data rows then point to the value in the lookup table. This can significantly reduce storage used when the data contain large repeating values, such as names. It is especially useful for materialized views with aggregation, since the data values for the grouping columns are often repeated.

To enable data segment compression you must specify the COMPRESS keyword when creating a tablespace, a table, or individual partitions of a range or list-partitioned table. It is not available for hash partitioned or index-organized tables. The following example shows a range-partitioned table, PURCHASES, whose first three partitions are compressed and the last one is left uncompressed.

CREATE TABLE easydw.purchases
(product_id varchar2(8),
 time_key date,
 customer_id varchar2(10),
 purchase_date date,
 purchase_time number(4,0),
 purchase_price number(6,2),
 shipping_charge number(5,2),
 today_special_offer varchar2(1))
PARTITION by RANGE (time_key)
 (partition purchases_jan2002
    values less than (TO_DATE('01-02-2002', 'DD-MM-YYYY'))
    pctfree 0 pctused 99
    storage (initial 64k next 64k pctincrease 0)
    tablespace purchases_jan2002 COMPRESS,
  partition purchases_feb2002
    values less than (TO_DATE('01-03-2002', 'DD-MM-YYYY'))
    pctfree 0 pctused 99
    storage (initial 64k next 64k pctincrease 0)
    tablespace purchases_feb2002 COMPRESS,
  partition purchases_mar2002
    values less than (TO_DATE('01-04-2002', 'DD-MM-YYYY'))
    pctfree 0 pctused 99
    storage (initial 64k next 64k pctincrease 0)
    tablespace purchases_mar2002 COMPRESS,
  partition purchase_catchall
    values less than (MAXVALUE)
    pctfree 0 pctused 99
    storage (initial 64k next 64k pctincrease 0)
    tablespace purchases_maxvalue NOCOMPRESS);

If you are adding a partition with segment compression for the first time to a previously uncompressed partitioned table, you need to rebuild any bitmap indexes on that table.

Hint:

To achieve better compression, when adding data into a new partition, sort the data first so that repeating values appear in the same block as much as possible.

You can change the compress attribute of a table by doing an ALTER TABLE MOVE command. For example, to compress the purchase_catchall partition of the purchases table, you issue:

ALTER TABLE purchases MOVE PARTITION purchases_catchall COMPRESS;

If new rows are added to a compressed table using a conventional INSERT statement, the new data is not compressed efficiently. Similarly, DELETE and UPDATE statements cannot maintain compression efficiently. However, if data is added using DIRECT PATH insert (using SQL Loader or an INSERT /*+APPEND */ statement), the new data is inserted into new blocks, which can be compressed. Hence, segment compression is well suited for data warehousing environments that do batch loads of data or for partitions that are mostly read only.

Hint:

You may want to merge and compress partitions corresponding to older data, since they are not likely to change very much. This allows you to keep more data online.


Query optimization techniques

3.5 Query optimization techniques

There are several different ways to retrieve any given row in the database. An index could be used to locate the row, or, if the table is small, it might be faster to perform a full table scan, reading each row sequentially. If there are multiple indexes, the optimizer chooses the best one to use. If there is a materialized view, the query may be rewritten to use it. The job of the optimizer is to choose a strategy to execute the query in the fastest possible time.

There are two approaches to query optimization:

  • Rule-based optimization

  • Cost-based optimization

The rule-based optimizer uses a set of rules to rank each of the possible different access paths available according to the speed of execution of each path. It chooses the access path with the best rank. The rule-based optimizer does not take into account the size of the table or the data distribution. Thus, it may always choose to use an index even though a full table scan may be more efficient.

The cost-based optimizer uses statistics, such as the cardinality of the table, number of distinct values of a column, and the data distribution, to determine the cost of an access path. It then picks the access path with the least cost to execute the query. The cost is a measure of how much I/O, CPU time, and memory will be required to execute the query. To use the cost-based optimizer effectively, statistics describing the cardinality and data distribution must be collected for each table, index, and materialized view. Chapter 7 will describe the use of the DBMS_STATS package to collect statistics.

Note that the rule-based optimizer is no longer being enhanced by Oracle. Many new features, such as partitioning, bitmap indexes, star query optimization, and query rewrite, are only available using the cost-based optimizer. Therefore, you should plan on using the cost-based optimizer for your data warehouse so you can take advantage of these features.

We will now look at various query optimizations used by the cost-based optimizer. Before we do so, let us take a brief look at the EXPLAIN PLAN facility used in several examples in this book.

3.5.1 EXPLAIN PLAN

Oracle provides several tools to display the execution plan. You can use the autotrace option of SQL*Plus to display the plan when you issue any query. Alternatively, to just get the query plan without executing the query, you can use EXPLAIN PLAN. The output of EXPLAIN PLAN is placed in a table known as PLAN_TABLE in your schema. You must create the PLAN_TABLE using the script $ORACLE_HOME/rdbms/admin/utlxplan.sql in your schema. You can also ask EXPLAIN PLAN to place the output in some other table, but it must have the same columns as the PLAN_TABLE. You can display the plan using scripts $ORACLE_ HOME/rdbms/admin/utlxpls.sql (for regular query plans) and utlxplp.sql (for parallel query plans).

Let us look at the output of an EXPLAIN PLAN statement. The output consists of the access path for each table, the cost of the access path, and the number of rows retrieved. The plan output is indented so that tables that are being joined are shown at the same level of indentation. Operations that are performed earlier are indented further.

EXPLAIN PLAN FOR
SELECT t.month, t.year, p.product_id,
       SUM (purchase_price) as sum_of_sales,
       COUNT (purchase_price) as total_sales,
       COUNT(*) as cstar
FROM time t, product p, purchases f
WHERE t.time_key = f.time_key AND
      f.product_id = p.product_id
GROUP BY t.month, t.year, p.product_id;

@utlxpls;

PLAN_TABLE_OUTPUT
------------------------------------------------------------------
Id|Operation              | Name    |Rows |Bytes|Cost|Pstart|Pstop
------------------------------------------------------------------
 0|SELECT STATEMENT       |         |  567|23814| 415|      |    |
 1| SORT GROUP BY         |         |  567|23814| 415|      |    |
 2|  NESTED LOOPS         |         |70585|2895K|  43|      |    |
*3|  HASH JOIN            |         |70585|2412K|  43|      |    |
 4|    TABLE ACCESS FULL  |TIME     |   15|  225|   2|      |    |
 5|    PARTITION RANGE ALL|         |     |     |    |   1  |  7 |
 6|     TABLE ACCESS FULL |PURCHASES|94113|1838K|  40|   1  |  7 |
*7|  INDEX RANGE SCAN     |PRODUCT_ |    1|    7|    |      |    |
  |                       | PK_INDEX|     |     |    |      |    |
------------------------------------------------------------------

Predicate Information (identified by operationid):
---------------------------------------------------
   3 - access("T"."TIME_KEY"="F"."TIME_KEY")
   7 - access("F"."PRODUCT_ID"="P"."PRODUCT_ID")

Note: cpu costing is off
21 rows selected.

In this example, the innermost operation is a hash join between the tables, PURCHASES, and TIME (rows 3-6). The output of this is joined with the PRODUCT table using a nested loops join (rows 2-7). Note that an index, PRODUCT_PK_INDEX (row 7), is used to access the PRODUCT table. The final operation is a sort in order to perform the GROUP BY. In Oracle 9i Release 2, the utlxpls.sql script has been enhanced to show the predicates being applied during each operation. In the previous example, the predicates being applied are the join predicates. The PARTITION RANGE clause (row 5) indicates the first and last partitions being scanned. The keyword ALL means that no partition pruning was done, causing all seven partitions to be scanned. Therefore, pstart = 1 and pstop = 7.

Now let us look at various techniques that the optimizer might use to return the data for our queries.

3.5.2 Join optimization

One of the most common operations in a query is a join between two tables. A join operation combines data from two or more tables, based on a condition (known as the join predicate) involving columns from the tables. There are three basic join methods used by the query optimizer in Oracle:

  • Sort-Merge join

  • Nested Loops join

  • Hash join

Sort-Merge join

In a sort-merge join, the data from each table is sorted by the values of the columns in the join condition. The sorted tables are then merged, so that each pair of rows with matching columns is joined. The SORT_AREA_ SIZE initialization parameter specifies the amount of memory available for sorting. Once this is exceeded, intermediate data is written to temporary segments in the TEMPORARY TABLESPACE on disk. If there is an index on the join columns of either table, the optimizer may use it to directly retrieve the data in sorted order, thereby avoiding the sort for that table.

Nested Loops join

In a nested loops join, one table is chosen as the outer table; the other is chosen as the inner table. For each row in the outer table, all matching rows that satisfy the join condition in the inner table are found. A nested loops join can be extremely efficient if the inner table has an index on the join column and there are few rows in the outer table.

Hash join

In a hash join, the smaller table is loaded into memory and a hashing function applied to the join columns, to build a hash table. Then, the second table is scanned; its join columns are hashed and compared with the hash table in memory to look for matching rows. If memory is not sufficient to hold the smaller table, the tables will be broken into smaller partitions. Hash joins can only be used for equi-joins (i.e., joins of the type table1.column1 = table2.column2). The HASH_AREA_SIZE specifies the amount of memory available for building the hash table before needing to write to disk.

Hash joins are especially useful when there are no indexes on the columns being joined. Given the ad hoc nature of decision-support queries, it is not always possible for the DBA to index all the columns the users may use to join tables together. When using Oracle as the staging area for data transformations, data may be moved from one temporary table into another. It may be more efficient to use a hash join than to take the time to build indexes on the tables. To enable a hash join, the HASH_JOIN_ ENABLED initialization parameter must be set to TRUE. If you use the Database Configuration Assistant tool to create a database for data warehousing, this parameter will automatically be enabled for you.

We will now look at some join optimizations that are of great value in a data warehouse.

Partition-wise join optimization

When the tables being joined are partitioned, the optimizer may choose to perform a partition-wise join. Rather than performing a large join between the tables, the join operation is broken up into a series of smaller joins between the partitions or subpartitions. Note that a partition-wise join can use any of the methods discussed previously to perform the join: sortmerge, hash, or nested loops joins.

A full partition-wise join can be done when the tables being joined are equi-partitioned on the join key in the query. Equi-partitioning means that there is a correspondence between the partitions (or subpartitions) of one table with the partitions (or subpartitions) of the other. So every partition of the first table must only be joined to its corresponding partition in the other table. When executing the query in parallel, each join can be executed on a separate processor.

If only one of the tables is partitioned on the join key, a partial partitionwise join can be done when executing the query in parallel. The table that is not partitioned by the join key is dynamically partitioned according to the partitioning criteria of the partitioned table, and each query server joins a pair of partitions from the two tables as in a full partition-wise join.

The decision to use partition-wise join or any of its variants is taken by the optimizer based on the cost of the execution plan.

Note that if the query only requires some of the partitions of any table, then only those partitions will participate in the partition-wise join.

Star Transformation

A star query is a typical query executed on a star schema. Each of the dimension tables is joined to the fact table using the primary-key/foreign-key relationship. If the fact table has bitmap indexes on the foreign-key columns, Oracle can optimize the performance of such star queries using an algorithm known as the star transformation. Star transformation is based on combining bitmap indexes on the fact table columns that correspond to the dimension tables in the query. First, bitmap indexes are used to retrieve the necessary rows from the fact table. The result is then joined to the dimension tables to get any columns required in the final answer. A bitmap join index may also be used instead of a combination of bitmap indexes. Star transformation is also possible if only some of the foreign-key columns have bitmap indexes.

If your data warehouse has a star schema, you should enable star transformation. This is done by setting the initialization parameter, STAR_ TRANSFORMATION_ENABLED, to TRUE. Alternatively, you can use the STAR_TRANSFORMATION hint on queries that join the fact and dimension tables. Note that star transformation is only available with the cost-based optimizer. The optimizer will weigh the execution plans, with and without star transformation, and pick the most efficient plan.

Suppose there were bitmap indexes on each of the foreign-key columns of the purchases table (i.e., purchases.customer_id, purchases.time_key, and purchases.product_id). The following example shows a query that can benefit from star transformation. This query joins the fact (purchases) table and dimension tables (product, time, customer) in a star schema and has selections on each of the dimension table as (month = 1 and county = 'Hants' and category = 'HDRW') representing a small fraction of the entire data.

SELECT c.town, t.quarter, p.product_name,
       SUM(f.purchase_price) sales
FROM purchases f, time t, customer c, product p
WHERE f.time_key = t.time_key and
      f.customer_id = c.customer_id and
      f.product_id = p.product_id and
      t.month = 1 and c.county = 'Hants' and
      p.category = 'HDRW'
GROUP BY c.town, t.quarter, p.product_name;

The execution plan is shown below.

------------------------------------------------------------------
|ID | Operation                       | Name    |Rows| Bytes|Cost|
------------------------------------------------------------------
|  0| SELECT STATEMENT                |         |  54|  3834| 715|
|  1|  SORT GROUP BY                  |         |  54|  3834| 715|
|* 2|   HASH JOIN                     |         | 328| 23288| 712|
|* 3|    TABLE ACCESS FULL            |PRODUCT  |  54|  1188|   2|
|* 4|    HASH JOIN                    |         | 984| 48216| 709|
|  5|     MERGE JOIN CARTESIAN        |         |   3|    78|   4|
|* 6|      TABLE ACCESS FULL          |CUSTOMER |   1|    12|   2|
|  7|      BUFFER SORT                |         |   1|    14|   2|
|* 8|       TABLE ACCESS FULL         |TIME     |   1|    14|   2|
|  9|     TABLE ACCESS BY INDEX ROWID |PURCHASES|7030|  157K| 703|
| 10|      BITMAP CONVERSION TO ROWIDS|         |    |      |    |
| 11|       BITMAP AND                |         |    |      |    |
| 12|        BITMAP MERGE             |         |    |      |    |
| 13|         BITMAP KEY ITERATION    |         |    |      |    |
|*14|          TABLE ACCESS FULL      |CUSTOMER |   1|    12|   2|
|*15|          BITMAP INDEX RANGE SCAN|CUST_IDX |    |      |    |
| 16|        BITMAP MERGE             |         |    |      |    |
| 17|         BITMAP KEY ITERATION    |         |    |      |    |
|*18|          TABLE ACCESS FULL      |TIME     |   2|    28|   2|
|*19|          BITMAP INDEX RANGE SCAN|TIME_IDX |    |      |    |
| 20|        BITMAP MERGE             |         |    |      |    |
| 21|         BITMAP KEY ITERATION    |         |    |      |    |
|*22|          TABLE ACCESS FULL      |PRODUCT  |  54|  1188|   2|
|*23|          BITMAP INDEX RANGE SCAN|PROD_IDX |    |      |    |
------------------------------------------------------------------

Let us try to understand how star transformation works. Notice that the rows corresponding to category = 'HDRW' can be retrieved using the subquery (this is also known as a semijoin):

SELECT *
FROM purchases
WHERE product_id IN (SELECT product_id
                    FROM product
                    WHERE category = 'HDRW');

This query can be executed using the prod_idx bitmap index, by retrieving and merging the bitmaps corresponding to all product_ids for the HDRW category. Similar subqueries are generated for each of the other dimension tables and, finally, the resulting bitmaps are combined using a bitmap AND operation into a single bitmap. This is shown in the execution plan in rows 10 through 23. This bitmap is used to retrieve the relevant rows from the purchases tables. Finally, a join is done to the dimension tables (shown in plan rows 0 through 9) to obtain the other column values (town, quarter, and product_name). This is efficient because only a small subset of the purchases table is now involved in the join.

Hint:

Creating a single column bitmap index for each foreign-key column in the fact table will allow the optimizer to consider star transformation for your star queries.


Automatic memory management

3.6 Automatic memory management

Queries used in business intelligence applications typically need a significant amount of memory for sorts, hash joins, bitmap operations, and so on. There are various initialization parameters, such as SORT_AREA_SIZE, HASH_AREA_SIZE, CREATE_BITMAP_AREA_SIZE, and BITMAP_ MERGE_AREA_SIZE, that must be tuned to get good performance. The ideal values for these parameters may vary from query to query and may depend on the load on the system. It is too time consuming for a DBA to tune these parameters accurately.

Oracle 9i introduced the automatic memory management feature to ease this burden. With this feature, the DBA only needs to specify the total amount of memory available for the instance by setting the PGA_ AGGREGATE_TARGET parameter. The optimizer determines the best way to utilize it for various SQL operations across all the database server processes running on that instance. To enable automatic memory management the initialization parameter WORKAREA_SIZE_POLICY must be set to AUTO (setting it to MANUAL will revert back to manual memory management).

The amount of memory allocated and used by each process can be seen in the V$PROCESS view, as follows:

select spid, pga_used_mem, pga_alloc_mem, pga_max_mem
from v$process;

SPID         PGA_USED_MEM PGA_ALLOC_MEM PGA_MAX_MEM
------------ ------------ ------------- -----------
340                132552        198880      198880
341                135056       3371624     3371624
343               4349880       7570468     7570468
...

The work area sizes used by each SQL statement are available in the V$SQL_WORKAREA. You can join to V$SQL to get the sql_text for the statements. (We have edited the following output to show only part of the sql_text for lack of space.)

SELECT sql_text, operation_type,
       estimated_optimal_size estsize, last_memory_used
FROM v$sql_workarea w, v$sql s
WHERE w.address = s.address
AND parsing_schema_id = USERENV('SCHEMAID') ;

SQL_TEXT        OPERATION_TYPE   ESTSIZE LAST_MEMORY_USED
--------------- --------------   ------- ----------------
SELECT t.month, GROUP BY (SORT)    56320            49152
SELECT t.month, HASH-JOIN         874496           628736
...

You can monitor the work area size in use while the query is executing in the V$SQL_WORKAREA_ACTIVE view, as follows:

-- shows the current workarea usage during an execution
SELECT operation_type, work_area_size, expected_size,
       actual_mem_used
FROM v$sql_workarea_active;

OPERATION_TYPE WORK_AREA_SIZE EXPECTED_SIZE ACTUAL_MEM_USED
-------------- -------------- ------------- ---------------
HASH-JOIN             1087488       1086464          628736


SQL functions

3.7 SQL functions

In this section, we will take a look at various SQL functions in Oracle.

3.7.1 Functions for aggregation

OLAP applications often involve aggregating data across several dimensions. To facilitate these operations, Oracle provides several aggregation capabilities. In addition to the traditional SQL GROUP BY clause, Oracle provides the CUBE, ROLLUP, and GROUPING SETS functions.

The CUBE and ROLLUP operators have been present since Oracle 8i, and the GROUPING SETS operator was introduced in Oracle 9i. The ROLLUP operator is used to compute subtotals along a dimension, and the CUBE operator is used to compute all possible aggregations for a set of GROUP BY columns. The GROUPING SETS operator enables a user to compute multiple independent groupings in one query. In the past, these types of operations were done using report-writing tools. By executing them within the database, they can be executed in parallel and benefit from the various features and query optimizations discussed earlier.

CUBE, ROLLUP, and GROUPING SETS can be used with all the supported aggregates in Oracle, such as COUNT, AVG, MIN, MAX, STDDEV, and VARIANCE and also with the analytical functions discussed in section 3.7.2.

We will now take a detailed look at these functions.

CUBE

The CUBE operator computes aggregates for all possible combinations of the columns in the GROUP BY clause. In the following example, we are computing the CUBE operation for total sales of products by product category and purchase date.

SELECT p.category, t.time_key, SUM(f.purchase_price)
FROM product p, purchases f, time t
WHERE p.product_id = f.product_id AND
   t.time_key = f.time_key
GROUP BY CUBE (p.category, t.time_key);

The answer to this query (shown below) includes SUM(f.purchase_ price) grouped by (category, time_key), by (category), by (time_key), and a grand total. We have highlighted with arrows the first row in each group.

CATEGORY   TIME_KEY             SALES
---------- --------------- ----------
ELEC       01-JAN-02       1574137.45   <- (category, time_key)
ELEC       02-JAN-02            56.02
ELEC       01-FEB-02            56.02
ELEC       02-FEB-02       1574137.45
HDRW       01-JAN-02         16704.22
HDRW       02-FEB-02         16704.22
MUSC       01-JAN-02         17989.16
MUSC       02-FEB-02         17989.16
ELEC                       3148386.94  <- (category)
HDRW                         33408.44
MUSC                         35978.32
           01-JAN-02       1608830.83  <- (time_key)
           02-JAN-02            56.02
           01-FEB-02            56.02
           02-FEB-02       1608830.83
                            3217773.7  <- grand total

ROLLUP

The ROLLUP function creates subtotals, rolling data up from a lower level to a higher level. It also produces a grand, or final, total. The ROLLUP operator is useful for totaling data across a hierarchical dimension such as time.

Referring to the following example, the ROLLUP operation computes the total sales for the groupings (category, time_key), (category), and grand total. We have highlighted the first row of each grouping with arrows.

SELECT p.category, t.time_key, SUM(f.purchase_price)
FROM product p, purchases f, time t
WHERE p.product_id = f.product_id AND
      t.time_key = f.time_key
GROUP BY ROLLUP (p.category, t.time_key);

CATEGORY   TIME_KEY             SALES
---------- --------------- ----------
ELEC       01-JAN-02       1574137.45   <- (category, time_key)
ELEC       02-JAN-02            56.02
ELEC       01-FEB-02            56.02
ELEC       02-FEB-02       1574137.45
HDRW       01-JAN-02         16704.22
HDRW       02-FEB-02         16704.22
MUSC       01-JAN-02         17989.16
MUSC       02-FEB-02         17989.16
HDRW                         33408.44   <- (category)
ELEC                       3148386.94
MUSC                         35978.32
                            3217773.7   <- grand total

In a ROLLUP, the groupings are determined by repeatedly aggregating over the rightmost column in the previous grouping. In the previous example, we first group by (category, time_key) and then by (category). So, to ROLLUP along a hierarchy correctly, we must order the columns from the highest to the lowest level of the hierarchy from left to right. For instance, to ROLLUP along a time hierarchy the column ordering would be (year, month, day).

If we compare the output of the CUBE and ROLLUP, you will notice that ROLLUP only computes some of the possible combinations of groupings in a CUBE. In the previous example, the grouping (time_key) is present in the CUBE but not in the ROLLUP output. The output of a CUBE always includes the output of a ROLLUP.

As the number of columns increases, the CUBE operator can consume a lot of time and space. The number of groupings for a CUBE with two columns is four; with three columns, eight; and so on. The ROLLUP is a much simpler and hence more efficient operation—for two columns, a ROLLUP produces three groupings; for three columns, four; and so on.

GROUPING SETS

Summary tables or materialized views are often used to store precomputed aggregations. If we were to store the result of a CUBE operator, computing all possible combinations of grouping columns, the space requirements could get too large. It is not uncommon for the output of a CUBE to be several times larger than the size of the tables themselves! This problem is overcome by using GROUPING SETS, which provide the capability to selectively compute only interesting combinations of groupings instead of the entire CUBE.

In the following example, we calculate sales by (category, time_key), by (category, state_id), by (time_key, region), and the grand total of sales, denoted by (). We choose not to calculate other combinations such as detailed sales for each category, time_key, and state.

SELECT p.category as cat, t.time_key, g.region, g.state_id as st,
       SUM(f.purchase_price) sales
FROM product p, purchases f, time t, geography g
WHERE p.product_id = f.product_id AND
      t.time_key = f.time_key AND
      g.state_id = f.state_id
GROUP BY GROUPING SETS ((p.category, t.time_key),
         (p.category, g.state_id), (t.time_key, g.region), ());

CAT   TIME_KEY   REGION    ST     SALES
----  ---------  --------- --  ---------
ELEC  01-JAN-02                1574137.45  <- (category,time_key)
ELEC  02-JAN-02                     56.02
ELEC  01-FEB-02                     56.02
ELEC  02-FEB-02                1574137.45
HDRW  01-JAN-02                  16704.22
HDRW  02-FEB-02                  16704.22
MUSC  01-JAN-02                  17989.16
MUSC  02-FEB-02                  17989.16

ELEC                       CA  1574193.47  <- (category,state_id)
ELEC                       MA  1484355.05
ELEC                       NH    89838.42
HDRW                       CA    13115.79
HDRW                       NH    16704.22
HDRW                       WI     3588.43
MUSC                       CA      2977.3
MUSC                       NH     7192.53
MUSC                       OH     7819.33
MUSC                       WI    17989.16

      01-JAN-02  West              2977.3   <- (time_key,region)
      01-JAN-02  MidWest          7819.33
      01-JAN-02  NorthEast      1598034.2
      02-JAN-02  NorthEast          56.02
      01-FEB-02  West               56.02
      02-FEB-02  West          1587253.24
      02-FEB-02  MidWest         21577.59

                               3217773.7   <- grand total

Note that ROLLUP is a special case of a GROUPING SET. ROLLUP(category, time_key) is equivalent to GROUPING SETS ((category, time_key), (category), ()).

You may specify multiple GROUPING SETS in a query. This offers a concise way to specify a cross-product of groupings across multiple dimensions. Suppose we would like to compute sales by product category on the geography dimension for the (state, city) columns and for the time dimension for the (year, quarter) columns. Instead of specifying all combinations involving these four columns, we could simply use the following query. This is known as concatenated GROUPING SETS.

SELECT p.category as cat, t.quarter as quart, t.year,
       g.state_id as st, g.region, SUM(f.purchase_price) sales
FROM purchases f, time t, geography g, product p
WHERE p.product_id = f.product_id AND
      t.time_key = f.time_key AND
      g.state_id = f.state_id
GROUP BY p.category,
         GROUPING SETS (g.state_id, g.region),
         GROUPING SETS (t.quarter, t.year);

CAT  QUART  YEAR   ST  REGION          SALES
---- -----  -----  --  ---------- ----------
ELEC     1         CA             1574193.47 <- (quarter,state)
ELEC     1         MA             1484355.05
ELEC     1         NH               89838.42
...
MUSC     1         WI               17989.16

ELEC     1             West       1574193.47 <- (quarter,region)
ELEC     1             NorthEast  1574193.47
...
MUSC     1             NorthEast     7192.53

ELEC        2002   CA             1574193.47  <- (year, state)
ELEC        2002   MA             1484355.05
ELEC        2002   NH               89838.42
...
MUSC        2002   WI               17989.16

ELEC        2002       West       1574193.47  <- (year, region)
ELEC        2002       NorthEast  1574193.47
...
MUSC        2002       NorthEast     7192.53

The p.category column that is outside the GROUPING SETS is present in all the groupings. Thus, the previous query computes the four groupings: (category, quarter, state), (category, quarter, region), (category, year, state), and (category, year, region).

GROUPING and GROUPING_ID functions

The CUBE, ROLLUP, and GROUPING SETS operators all compute multiple levels of aggregations in one query. Your application may need a way to identify the portion of the output corresponding to a particular level of aggregation. The SQL functions, GROUPING() and GR OUPING_ID() provide a mechanism to identify which rows in the answer correspond to which level of aggregation.

The following query illustrates the behavior of the GROUPING() function.

SELECT p.category as cat, t.time_key,
       SUM(f.purchase_price) sales,
       GROUPING(p.category) grp_c, GROUPING(t.time_key) grp_t
FROM product p, purchases f, time t
WHERE p.product_id = f.product_id AND
      t.time_key = f.time_key
GROUP BY ROLLUP (p.category, t.time_key);

CAT   TIME_KEY        SALES GRP_C GRP_T
----  ---------  ---------- ----- -----
ELEC  01-JAN-02  1574137.45     0     0   <- (category,time_key)
ELEC  02-JAN-02       56.02     0     0
ELEC  01-FEB-02       56.02     0     0
ELEC  02-FEB-02  1574137.45     0     0
HDRW  01-JAN-02    16704.22     0     0
HDRW  02-FEB-02    16704.22     0     0
MUSC  01-JAN-02    17989.16     0     0
MUSC  02-FEB-02    17989.16     0     0
HDRW                4455.58     0     0 *
ELEC             3148386.94     0     1   <- (category)
HDRW               37864.02     0     1 *
MUSC               35978.32     0     1
                 3222229.28     1     1   <- grand total

For each grouping, the GROUPING(category) returns a value of 0 if the category column is in the group and 1 otherwise. Similarly, GROUPING(time_key) returns a value of 0 if the time_key column is in the group and 1 otherwise. Thus, each level of aggregation can be identified from the values of the GROUPING() function. The group (category, time_key) has GROUPING function values (0,0); the group (category) has GROUPING function values (0,1). Note that the grand total row can be easily identified as the row where each grouping function column has the value of 1.

In the output of CUBE or ROLLUP, the rows that correspond to a higher level of aggregation have the value NULL for the columns that have been aggregated away. Another use of the GROUPING() function is to distinguish this NULL from actual NULL values in the data. For example, look carefully at the rows in the previous output marked with an asterisk. Both these rows have a NULL value in the time_key column. The first of these corresponds to rows in the purchases table for the HDRW category, where the time_key value was unavailable (NULL). In the second one, we have aggregated away the time_key values and hence this row corresponds to aggregation at the category level. The GROUPING() function distinguishes these two similar-looking rows. For the first case, the GROUPING(time_key) is 0; in the second, it is 1.

Instead of using GROUPING() for each column, you can use GROUPING_ID() with all the columns together, as follows:

SELECT p.category, t.time_key, SUM(f.purchase_price),
       GROUPING_ID(p.category,t.time_key) gid
FROM product p, purchases f, time t
WHERE p.product_id = f.product_id AND
    t.time_key = f.time_key
GROUP BY ROLLUP (p.category, t.time_key);

CATEGORY TIME_KEY       SALES        GID
-------- --------- ---------- ----------
ELEC     01-JAN-02 1574137.45          0    <- (category, time_key)
ELEC     02-JAN-02      56.02          0
ELEC     01-FEB-02      56.02          0
ELEC     02-FEB-02 1574137.45          0
HDRW     01-JAN-02   16704.22          0
HDRW     02-FEB-02   16704.22          0
MUSC     01-JAN-02   17989.16          0
MUSC     02-FEB-02   17989.16          0
HDRW                  4455.58          0
ELEC               3148386.94          1     <- (category)
HDRW                 37864.02          1
MUSC                 35978.32          1
                   3222229.28          3     <- grand total

If you consider all the individual GROUPING() functions together as a binary number, GROUPING_ID() returns the corresponding decimal number. Thus, in the previous example, if GROUPING(p.category) is 0 and GROUPING(t.time_key) is 1, then GROUPING_ID(p.category, t.time_key) is the binary number formed by 01, which is the decimal number 1. For the grand total row, GROUPING(p.category) is 1 and GROUPING(t.time_key) is 1, and, hence, the GROUPING_ID(p.category, t.time_key) is the binary number 11, which is the number 3. The GROUPING_ID() is thus a much more compact representation than individual grouping functions but is not as straightforward to interpret as separate GROUPING() functions on each column.

3.7.2 SQL functions for analytical computing

Decision support applications often need to answer questions such as: What were the top-ten selling products in 2001? or How do the regional sales for January this year compare against last year's sales? or What are the cumulative sales numbers for each month this year?

Though these questions sound quite simple, the traditional SQL queries needed to answer these questions can be extremely complex. These queries are quite hard to optimize and may require multiple scans over the data. They may perform poorly or may require application layer processing. Hence, many database vendors started an initiative to provide SQL extensions to concisely represent these types of queries. The extensions are now part of the SQL99 standard. By using the analytical functions in the database, these calculations can take advantage of parallelism and other optimization techniques in the database.

The analytical functions in Oracle provide many useful capabilities, including the following:

  • Ranking functions can be used to answer queries for top-N items. They can also be used to classify data into buckets. Examples of ranking functions include RANK(), DENSE_RANK(), and NTILE(). You can also compute the rank of a quantity as if it were hypothetically inserted into the data.

  • Moving window functions can be used to calculate quantities such as cumulative sum or moving average. Moving window calculations are very useful when dealing with data over a period of time. For instance, if you want to know the average sales of a product over the past seven days, six months, five years, and so on.

  • Reporting aggregates can be used to see the aggregate value side by side with the detailed rows that contributed to it. For instance, if you want to compare sales of each product with the average sales of all products.

  • Lag and lead functions can be used to do period-over-period comparisons—for example, comparing sales of one year with the previous.

  • Statistical functions, such as linear regression, and advanced aggregate functions, such as covariance, can be used to do trend analysis.

We will now look at these functions in detail with examples.

Ranking functions

Ranking functions rank the rows in a data set after ordering it by a specified criterion. The following SQL example answers the query: What were the top-selling ten products? In this case, the ordering criterion is the total sales for the product, as specified by the ORDER BY clause.

SELECT *
FROM
(SELECT p.product_id, SUM(f.purchase_price) as sales,
        RANK() over (ORDER BY SUM(f.purchase_price)) as rank
 FROM purchases f, product p
 WHERE f.product_id = p.product_id
 GROUP BY p.product_id)
WHERE rank <= 10;

PRODUCT_ID      SALES      RANK
---------- ---------- ---------
SP1222       38832.31         1
SP1244       39145.54         2
SP1256       39354.24         3
SP1263       39458.78         4
SP1236       39699.14         5
SP1220       39893.6          6
SP1231       40080.61         7
SP1242       40208.33         8
SP1248       40312.98         9
SP1254       40418.02        10

This query has two parts—an inner query that computes the RANK() function and the outer one that selects the top ten of these ranks. The inner query computes the aggregate function SUM(f.purchase_price) for each product_id. It then orders the result by the sales and applies the RANK() function to it. The RANK() function assigns ranks from 1 to N, skipping ranks in case of ties. Thus, if there were two products with the same sales, with rank 3, the next rank would be 5. A variant of RANK() is the DENSE_RANK() function which assigns contiguous ranks despite ties. For instance, if two products had the same rank, 3, DENSE_RANK() would assign the next rank to be 4.

The ORDER BY clause further specifies ascending versus descending order and whether to consider NULL values as first or last in the order. The default order is ascending. NULLS LAST is the default for ascending order and NULLS FIRST for descending order. In the following query, the ORDER BY is in descending order and NULL values are to be considered last in the order.

SELECT p.product_id, SUM(f.purchase_price) as sales,
       RANK() over (ORDER BY SUM(f.purchase_price) DESC NULLS LAST)
       as rank
 FROM purchases f, product p
 WHERE f.product_id = p.product_id
 GROUP BY p.product_id;

PRODUCT_      SALES       RANK
-------- ---------- ----------
SP1046   1115634.20          1
SP1056   1109091.08          2
SP1036   1107147.52          3
SP1060   1095253.99          4
SP1042   1002747.91          5
SP1051   979279.86           6
SP1041   941985.49           7
SP1054   919424.74           8
SP1048   919051.12           9
SP1052   853124.58          10
SP1222   81385.09          161
SP1039                     162   <- nulls last

It is important to note that analytical functions are applied after the WHERE, GROUP BY, and HAVING clauses of the query are computed. So aggregate functions such as SUM(f.purchase_price) are available as ordering criteria to the ranking function.

PARTITION BY clause

A useful feature with analytic functions is that you can divide the data into groups and then apply the function to each group. The group is specified using the PARTITION BY clause. For instance, if you wanted to rank the products in each product category, we could use the following query. (Only five ranks are shown in each category for lack of space.)

SELECT p.category, p.product_id, SUM(f.purchase_price) as sales,
       RANK() over (PARTITION BY p.category
                    ORDER BY SUM(f.purchase_price)) as rank
FROM purchases f, product p
WHERE f.product_id = p.product_id
GROUP BY p.category, p.product_id;
CATEGORY   PRODUCT_ID      SALES       RANK
---------- ---------- ---------- ----------
ELEC       SP1080        40495.3          1
ELEC       SP1073       41440.34          2
ELEC       SP1086       41599.05          3
ELEC       SP1065       41939.38          4
ELEC       SP1084       42253.72          5
...
HDRW       SP1222       38832.31          1
HDRW       SP1236       39699.14          2
HDRW       SP1220        39893.6          3
HDRW       SP1231       40080.61          4
HDRW       SP1214       40930.96          5
...

Figure 3.9 illustrates how the PARTITION BY clause affects the computation of RANK(). As before, the query first computes the sales for each product_id and category. The PARTITION BY clause in the query then divides this result by category. For each category, the RANK() function assigns a rank to the product_ids ordered according to their sales.

Click To expand
Figure 3.9: PARTITION BY clause.

You can calculate ranks using different PARTITION and ORDER BY expressions in the same query. The PARTITION BY clause used by analytical functions is completely unrelated to the data partitioning feature of Oracle discussed in section 3.3.

Bucketing functions

Analytical applications often require the ability to classify data into buckets or to determine how a given product compares in the entire range of products. The ranking functions CUME_DIST(), PERCENT_RANK(), and NTILE() provide this functionality.

CUME_DIST() calculates the position of a value in relation to a set of values, returning a value between 0 and 1. PERCENT_RANK() gives the rank of a row relative to all the rows in a data set Both CUME_DIST() and PERCENT_RANK() have a value between 0 and 1.

The following examples illustrate the difference between CUME_DIST and PERCENT_RANK functions. (Only rows for the HDRW category are shown.)

SELECT p.category, p.product_id, SUM(f.purchase_price) as sales,
       CUME_DIST() over (PARTITION BY p.category
                       ORDER BY SUM(f.purchase_price)) as cume_dist
FROM purchases f, product p
WHERE f.product_id = p.product_id
GROUP BY p.category, p.product_id;
HAVING SUM(f.purchase_price) < 100000 ;

CATE PRODUCT_      SALES  CUME_DIST
---- -------- ---------- ----------
...
HDRW SP1222     81385.09 .142857143
HDRW SP1236     82520.97 .285714286
HDRW SP1220        82622 .428571429
HDRW SP1231     83073.57 .571428571
HDRW SP1214     83841.66 .714285714
HDRW SP1212     99663.51 .857142857
HDRW SP1232     99863.42          1
...

The position of the first row is 1 and there are 7 rows in all, so CUME_ DIST() is 1/7, or 0.142857143, and so on. The PERCENT_RANK() is computed as RANK() of the row - 1, divided by total ranks - 1. Thus, PERCENT_RANK() for the first row is always 0. The RANK() of the second row is 2 and there are 7 ranks, so PERCENT_RANK is 1/6, or 0.166666667.

SELECT p.category, p.product_id, SUM(f.purchase_price) as sales,
       PERCENT_RANK() over (PARTITION BY p.category
                            ORDER BY SUM(f.purchase_price))
                            as pct_rank
FROM purchases f, product p
WHERE f.product_id = p.product_id
GROUP BY p.category, p.product_id
HAVING SUM(f.purchase_price) < 100000;

CATE PRODUCT_      SALES  CUME_DIST
---- -------- ---------- ----------
...
HDRW SP1222     81385.09          0
HDRW SP1236     82520.97 .166666667
HDRW SP1220        82622 .333333333
HDRW SP1231     83073.57         .5
HDRW SP1214     83841.66 .666666667
HDRW SP1212     99663.51 .833333333
HDRW SP1232     99863.42          1
...

The NTILE() function is useful to compute statistical measures such as quartile and percentile. The NTILE(N) function orders the data using specified criteria and then divides the result into N buckets, assigning the bucket number to all rows in each bucket. If N is 100, the quantity is known as the percentile and if N is 4, quartile.

The following query calculates the NTILE(4) in each product category using the sales for each product in the category. NTILE(4) assigns a number between one and four to each row.

SELECT p.category, p.product_id, SUM(f.purchase_price) as sales,
 NTILE(4) over (PARTITION BY p.category
                ORDER BY SUM(f.purchase_price)) as quartile
FROM purchases f, product p
WHERE f.product_id = p.product_id
GROUP BY p.category, p.product_id;

CATE PRODUCT_      SALES  CUME_DIST
---- -------- ---------- ----------
...
HDRW SP1222     81385.09          1
HDRW SP1236     82520.97          1
HDRW SP1220        82622          2
HDRW SP1231     83073.57          2
HDRW SP1214     83841.66          3
HDRW SP1212     99663.51          3
HDRW SP1232     99863.42          4
...

The buckets generated by the NTILE(4) function all have approximately the same number of rows; however, the range between lowest and highest value in each bucket may differ. This is known as an equi-height histogram. On the other hand, the WIDTH_BUCKET() function can be used to generate equi-width histograms where the data is distributed into buckets such that each bucket has the same range but can have differing numbers of rows.

SELECT p.product_id, SUM(f.purchase_price) as sales,
       WIDTH_BUCKET(SUM(f.purchase_price), 80000, 120000, 4)
          as width_bucket
FROM purchases f, product p
WHERE f.product_id = p.product_id and p.category = 'HDRW'
GROUP BY p.product_id
ORDER BY sales;

PRODUCT_      SALES WIDTH_BUCKET
-------- ---------- ------------
SP1222     81385.09            1
SP1236     82520.97            1
SP1220     82622.00            1
SP1231     83073.57            1
SP1214     83841.66            1
SP1212     99663.51            2
SP1232     99863.42            2
SP1224    100825.37            3
SP1234    109433.07            3
SP1219    111809.95            4
SP1239    112749.64            4
SP1213    112749.90            4
...
SP1233    148895.43            5
SP1221    167743.34            5

The inputs to the WIDTH_BUCKET are the end points for the ranges to be considered and the number of intermediate points. The function then divides the range of sales into the equi-width buckets by placing equally spaced points in between the end points and assigns each row to buckets according to which range it falls into. In our examples the range from 80000 to 120000 is divided into five approximately equal ranges (or five buckets). We can see that each bucket can have different numbers of rows.

Window aggregate functions

Window aggregate functions let you compute a function such as SUM over a specified window of rows relative to the current row. These functions allow you to do moving window operations such as cumulative sum or moving average.

The following SQL statement answers the query: What are the cumulative sales numbers for each month so far this year? The query first computes the SUM(f.purchase_price) for each month. The expression SUM(SUM(f.purchase_price)) indicates that we are further summing the sales, in this case for all months preceding the current month. The ROWS UNBOUNDED PRECEDING specifies that the window is all rows before and including the current row. A window specified using numbers of rows preceding or following the current row is known as a physical window.

SELECT t.month, SUM(f.purchase_price) as sales,
       SUM(SUM(f.purchase_price))
       OVER (ORDER BY t.month ROWS UNBOUNDED PRECEDING)
          as cumulative_sales
FROM purchases f, time t
WHERE f.time_key = t.time_key
GROUP BY t.month;

MONTH        SALES  CUMULATIVE_SALES
------  ----------  ----------------
1       8989096.34        8989096.34
2      17213413.50       26202509.90
3        429035.78       26631545.70
4       4520301.94       31151847.60
5       1268317.48       32420165.10
6       7092585.46       39512750.50
...

In the next example, we are computing, for each month, the moving average of the sales for that month and the two months preceding it. This is specified by the ROWS 2 PRECEDING clause.

SELECT t.month, SUM(f.purchase_price) as sales,
       AVG(SUM(f.purchase_price))
       OVER (ORDER BY t.month ROWS 2 PRECEDING) as moving_average
FROM purchases f, time t
WHERE f.time_key = t.time_key
GROUP BY t.month;

MONTH           SALES  MOVING_AVERAGE
---------- ----------  --------------
1          8989096.34      8989096.34
2         17213413.5      13101254.90
3           429035.78      8877181.89
4          4520301.94      7387583.75
5          1268317.48      2072551.73
6          7092585.46      4293734.96
...

If, instead, we wanted the window to include the current month and two months following it, the window expression would simply change to ROWS 2 FOLLOWING. A physical window is good only if you have dense data—for instance, if you have gaps in your dates, a physical window may not be very meaningful.

Instead of specifying the offset using rows, you can specify a logical window using an interval of values around the current value. For instance, if the ordering expression is of DATE type, you can specify a logical window using a constant expression or an INTERVAL DAY, MONTH, or YEAR expression. Logical windows are only allowed for numeric, date, or interval data types. The following example shows sales being computed over a five day interval including two days before and after the current date. In our example, the interval for the dates 2-Apr-02 consists of dates from 31-Mar-02 through 4-Apr-02, and that for 7-Jul-02 consists of dates from 5-Jul-02 through 8-Jul-02, as shown in the answer.

SELECT t.time_key, SUM(f.purchase_price) as sales,
       SUM(SUM(f.purchase_price)) OVER (ORDER BY t.time_key
       RANGE BETWEEN INTERVAL '2' DAY PRECEDING AND
                     INTERVAL '2' DAY FOLLOWING) as sales_5_day
FROM purchases f, time t
WHERE f.time_key = t.time_key
GROUP BY t.time_key;

TIME_KEY        SALES  SALES_5_DAY
---------  ----------  -----------
01-JAN-02  8989040.32   8989096.34
02-JAN-02       56.02   8989096.34
01-FEB-02  3401896.62   17213525.6
02-FEB-02    13811629   17213525.6
02-MAR-02   429035.78    429035.78
30-MAR-02      448.19   1958695.44                   _
31-MAR-02      448.19   1960040.01                    |
01-APR-02  1957799.06   1960936.39                    |
02-APR-02     1344.57   1962280.96   <- current row   | logical
03-APR-02      896.38   1961832.77                    | interval
04-APR-02     1792.76      4033.71                   _|
08-APR-02  2562502.88   2562502.88
06-MAY-02    839281.7   1268317.48
07-MAY-02   429035.78   1268317.48
01-JUN-02  6192443.48   7092585.46
03-JUN-02   900141.98   7092585.46
02-JUL-02       56.02        56.02                   _
05-JUL-02     2240.95      2268.96                    |
07-JUL-02       28.01      2296.97   <-  current row  | logical
08-JUL-02       28.01        56.02                   _| interval

The previous example used the BETWEEN clause to specify upper and lower bounds for the interval. This can also be used with a physical offset (e.g., ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING).

Reporting aggregates

Suppose we want to answer the question: What is the best-selling product category for each sales region? If we try to write a SQL query for this, we will quickly find that the SQL required to answer such a simple question is very complex. For example, one way of doing it would be to first create a table, say total_sales, with the total sales by product category for each sales region. Then further aggregate this table to find the maximum sales for each region, stored in tables, max_sales. Then join the total_sales table to the max_sales table to find those rows that correspond to the maximum sales for each region and return the corresponding product categories. You could also write all these steps in one big SQL query involving several subqueries, but it is quite cumbersome.

The reason why these types of questions are so complex is that when you ask for a simple aggregate such as SUM or MAX in SQL, you lose the individual rows contributing to the aggregate. You only get one row back, which is the aggregate. So when we computed the maximum sales in the max_sales table, we lost knowledge about the individual product categories that had this value of sales.

Reporting aggregates solve this problem by reporting the computed aggregate value side by side with all the detail rows that contributed to it. As with other analytical functions, you can use the PARTITION BY clauses to divide the data before computing the reporting aggregate.

The following query finds the maximum sales for each region, using a reporting aggregate. The PARTITION BY clause simply divides the data by region. The reporting aggregate MAX(SUM(f.purchase_price)) computes the maximum of sales for each region and reports it for each product category value. Thus, the value of max_sales for the region is repeated for each product category in each region. The way we identify that it is a reporting aggregate and not a regular aggregate is the OVER clause.

SELECT p.category, g.region,
       SUM(f.purchase_price) as regional_prod_sales,
       MAX(SUM(f.purchase_price)) OVER (PARTITION BY g.region)
           as max_sales
FROM product p, geography g, purchases f
WHERE f.product_id = p.product_id
  AND f.state_id = g.state_id
GROUP BY p.category, g.region;
CATEGORY   REGION     REGIONAL_PROD_SALES       MAX_SALES
---------- ---------- -------------------      ----------
HDRW       MidWest                8044.01        25808.49
MUSC       MidWest               25808.49        25808.49   *
ELEC       NorthEast           1574193.47      1574193.47   *
HDRW       NorthEast             16704.22      1574193.47
MUSC       NorthEast              7192.53      1574193.47
ELEC       West                1574193.47      1574193.47   *
HDRW       West                  13115.79      1574193.47
MUSC       West                    2977.3      1574193.47

Now to answer our original question: What is the best-selling product category for each sales region? All we need to do is select those rows from the previous answer where the regional_prod_sales is the same as the max_ sales. These rows are marked with an asterisk in the preceding output.

SELECT category, region, regional_prod_sales
FROM
(
 SELECT p.category, g.region,
        SUM(f.purchase_price) as regional_prod_sales,
        MAX(SUM(f.purchase_price))
            OVER (PARTITION BY g.region) as max_sales
FROM product p, geography g, purchases f
WHERE f.product_id = p.product_id
AND f.state_id = g.state_id
GROUP BY p.category, g.region
)
WHERE regional_prod_sales = max_sales;

CATEGORY   REGION     REGIONAL_PROD_SALES  MAX_SALES
---------- ---------- -------------------  ----------
MUSC       MidWest               25808.49    25808.49
ELEC       NorthEast           1574193.47  1574193.47
ELEC       West                1574193.47  1574193.47

Another example where reporting aggregates come in handy is to report the salary of each employee along with the average salary in the department.

SELECT employee, department, salary
       AVG(salary)
          OVER (PARTITION by department) as avg_dept_salary
FROM payroll;

A special case of a reporting aggregate is the RATIO_TO_REPORT function, which computes the ratio of a value in a set to the sum of the values in that set. The following example finds the ratio of the sales in each product category to the total sales for all product categories. SUM(SUM(f.purchase_price)) computes the total sales for each product category. RATIO_TO_REPORT is essentially a short form for dividing reg_sales/reg_sales_total.

SELECT category, SUM(f.purchase_price) reg_sales,
  SUM(SUM(f.purchase_price)) OVER () as reg_sales_total,
  RATIO_TO_REPORT(SUM(f.purchase_price)) OVER () as ratio_to_report
FROM product p, purchases f
WHERE f.product_id = p.product_id
GROUP BY p.category;

CATE   REG_SALES  REG_SALES_TOTAL RATIO_TO_REPORT
----  ----------- --------------- ---------------
ELEC  33334833.10      39520369.7      .843484848
HDRW   2962332.61      39520369.7      .074957108
MUSC   3223204.06      39520369.7      .081558044

An interesting aspect of the previous example is that it uses an empty OVER() clause. This simply means that the function is being computed without any partition by clause (i.e., over all rows).

Lag and lead functions

Lag and lead functions provide access to values of a column from a row at a known offset from the current row. For instance, Lag(sales, 1) gives the value of the sales column in the previous row. Lead(sales, 1) gives the value of the sales column in the next row. Thus, queries that would otherwise need to join a table to itself can be done without the join. For instance, the following query reports the sales by month alongside the sales for the previous month and the following month for the year 2002.

SELECT t.month, SUM(f.purchase_price) as sales,
       LAG(SUM(f.purchase_price),1)
            OVER (ORDER BY t.month) as sales_last_month,
       LEAD(SUM(f.purchase_price),1)
            OVER (ORDER BY t.month) as sales_next_month
FROM purchases f, time t
WHERE f.time_key = t.time_key AND t.year = 2002
GROUP BY t.month;

     MONTH        SALES   SALES_LAST_MONTH   SALES_NEXT_MONTH
----------   ----------   -----------------  ----------------
         1   8989096.34                            17213413.5
         2  17213413.5           8989096.34         429035.78
         3    429035.78         17213413.5         4520301.94
         4   4520301.94           429035.78        1268317.48
         5   1268317.48          4520301.94        7092585.46
         6   7092585.46          1268317.48
...

Lag and lead could be used to do period over period comparisons of sales. For instance, to compare sales for a quarter to the same quarter a year ago (i.e., four quarters ago), you could use LAG(sales, 4).

Note that if the previous (or next) row is not present, Lag (Lead) returns the NULL value.

FIRST and LAST functions

In some situations, you may need to order data by a certain quantity and do analysis based on the first and last items in that order. The FIRST and LAST aggregation functions allow you to do such operations. For instance, the following query finds the number of purchases made for the costliest and cheapest products in each category. It first orders the products in each category by their sell_price and then aggregates the rows obtained using COUNT(*) to find numbers of purchases made for those products that rank FIRST or LAST. Note that the aggregate serves as a tiebreaker if there are several candidates for the FIRST or LAST place by generating one row for all of them.

SELECT p.category cat, SUM(f.purchase_price) total_sales,
       MIN(p.sell_price) cheap_prod,
       COUNT(*) KEEP (DENSE_RANK FIRST
                      ORDER BY p.sell_price) cheap_sales,
       MAX(p.sell_price) costly_prod,
       COUNT(*) KEEP (DENSE_RANK LAST
                      ORDER BY p.sell_price) costly_sales
FROM purchases f, product p
WHERE f.product_id = p.product_id
GROUP BY p.category;

CAT  TOTAL_SALES CHEAP_PROD CHEAP_SALES COSTLY_PROD COSTLY_SALES
---- ----------- ---------- ----------- ----------- ------------
ELEC  33327213.9      24.99       16141     1267.89        15036
HDRW  2962332.61      15.67        9980       18            5001
MUSC  3223204.06      10           2305       15.67        13710

FIRST and LAST functions can be used to find sales trends—for instance, in our example we find that in the ELEC category, sales for the costliest items are about the same as for the cheaper ones, whereas in the HDRW category, about a $3 price difference almost cuts the purchases by half.

The FIRST and LAST functions can also be used as reporting aggregates.

Statistical analysis functions

Business decisions may often be influenced by relationships between various quantities. For instance, the price of an item may influence how many items were sold. Linear regression analysis is a statistical technique used to quantify how one quantity affects or determines the value of another. The idea is to fit the data for two quantities along a straight line as accurately as possible. This line is called the regression line. Some of the functions of interest are the slope of the line, y-intercept of the line, and the coefficient of determination (which is how closely the line fits the points). Oracle provides various diagnostic functions commonly used for this analysis, such as standard error and regression sum of squares.

The linear regression functions are all computed simultaneously in a single pass through the data. They can be treated as regular aggregate functions or reporting aggregate functions.

The following example illustrates the use of some of these functions. Here, we are computing the slope, intercept, and coefficient of determination of the regression line for sell_price and purchase_price for products in each product category.

SELECT p.category cat,
       REGR_SLOPE(p.sell_price, f.purchase_price) slope,
       REGR_INTERCEPT(p.sell_price, f.purchase_price) intercept,
       REGR_R2(p.sell_price, f.purchase_price) coeff_determination
FROM purchases f, product p
WHERE f.product_id = p.product_id
GROUP BY p.category;

CAT        SLOPE  INTERCEPT COEFF_DETERMINATION
---- ----------- ---------- -------------------
ELEC  .830878246  -87.405372         .831264004
HDRW  -.00003465  16.4546582         .000057048
MUSC  -.00005981  14.8659674         .000053216

From this analysis, we can see that the relationship between sell_price and purchase_price for the ELEC category can be closely modeled by a straight line. However, for the HDRW and MUSC category, the line is not a very good fit.

To support linear regression analysis, Oracle also provides some new aggregate functions to compute covariance of a population (COVAR_POP) or sample (COVAR_SAMP) and correlation (CORR) between variables.

Inverse percentile

Previously, we mentioned the CUME_DIST() function, which determines the position of each row of data with respect to the entire set. This is also known as a percentile. An inverse percentile performs the reverse operation (i.e., returns the data value that corresponds to a given percentile value in an ordered set of rows).

The following query computes the CUME_DIST() of sales for each category. Here the rows are ordered by SUM(purchase_price), and each row is assigned a value between 0 and 1. We only show some of the rows.

SELECT p.category, p.product_id, SUM(f.purchase_price) as sales,
    CUME_DIST() over (PARTITION BY p.category
                      ORDER BY SUM(f.purchase_price)) as cume_dist
FROM purchases f, product p
WHERE f.product_id = p.product_id
GROUP BY p.category, p.product_id;

CATE PRODUCT       SALES  CUME_DIST
---- -------- ---------- ----------
...
ELEC SP1100    150155.8  .495412844
ELEC SP1093    150337.4  .504587156   <- PERCENTILE_DISC(0.5)
ELEC SP1076    150717.89 .513761468
...
HDRW SP1213    112749.9  .461538462
HDRW SP1217    113019.02         .5   <- PERCENTILE_DISC(0.5)
HDRW SP1122    113648.09 .538461538
...
MUSC SP1251    114592.31 .481481481
MUSC SP1253    114840.43 .518518519   <- PERCENTILE_DISC(0.5)
MUSC SP1264    115221.38 .555555556
...

There are two flavors of inverse percentile—PERCENTILE_DISC() assumes that the values are discrete and returns the data value that corresponds to the nearest CUME_DIST() value greater than the percentile specified. PERCENTILE_CONT() assumes that the values are continuous and returns the interpolated value corresponding to the given percentile. In the previous example, the sales value that corresponds to PERCENTILE_ DISC(0.5) for the HDRW category is 113019.02 and for MUSC category is 114840.43.

The following example illustrates the use of the inverse percentile functions. Note that since the PERCENTILE_DISC and PERCENTILE_ CONT functions must return a single data value, an ORDER BY criterion must be specified and must consist of a single expression.

SELECT p.category, p.product_id, SUM(f.purchase_price) as sales,
       PERCENTILE_DISC(0.5) WITHIN GROUP
                   (ORDER BY SUM(f.purchase_price))
                   OVER (PARTITION BY p.category) as pct_disc,
       PERCENTILE_CONT(0.5) WITHIN GROUP
                   (ORDER BY SUM(f.purchase_price))
                   OVER (PARTITION BY p.category) as pct_cont
FROM purchases f, product p
WHERE f.product_id = p.product_id
GROUP BY p.category, p.product_id;

CATE PRODUCT_      SALES   PCT_DISC   PCT_CONT
---- -------- ---------- ---------- ----------
...
ELEC SP1100     150155.8   150337.4   150337.4
ELEC SP1093     150337.4   150337.4   150337.4
ELEC SP1076    150717.89   150337.4   150337.4
...
HDRW SP1213     112749.9  113019.02 113333.555
HDRW SP1217    113019.02  113019.02 113333.555
HDRW SP1122    113648.09  113019.02 113333.555
...
MUSC SP1251    114592.31  114840.43  114840.43
MUSC SP1253    114840.43  114840.43  114840.43
MUSC SP1264    115221.38  114840.43  114840.43
...

In this example, PERCENTILE_DISC and PERCENTILE_CONT have been used as reporting aggregates.

What-if analysis

Analytical applications often involve what-if analysis. For instance, you may want to know how a new product will perform relative to other products in its category, given a projected sales figure. Oracle provides a family of hypothetical rank and distribution functions for this purpose. With these functions, you can ask to compute the RANK(), PERCENT_RANK(), or CUME_DIST() of a given value, if it were hypothetically inserted into a set of values.

Hypothetical rank functions take an ordering condition and a constant data value to be inserted into the ordered set. To illustrate we show the sales for different products and then find the rank for a hypothetical product.

The following query shows the sales for different products in the HDRW category and their respective ranks.

SELECT p.product_id, SUM(f.purchase_price) sales,
       RANK() OVER (ORDER BY SUM(f.purchase_price)) as rank
FROM purchases f, product p
WHERE f.product_id = p.product_id and p.category = 'HDRW'
GROUP BY p.product_id;

CATE PRODUCT_      SALES  RANK
---- -------- ---------- -----
HDRW SP1222     81385.09     1
HDRW SP1236     82520.97     2
HDRW SP1220        82622     3
HDRW SP1231     83073.57     4
HDRW SP1214     83841.66     5
                              <- insert hypothetical value 98000.00
HDRW SP1212     99663.51     6
HDRW SP1232     99863.42     7
HDRW SP1224    100825.37     8
...

Now, suppose we want to find the hypothetical rank of a product with sales of $98,000.00. From the previous output, we can see that this value, if inserted into the data, would get a rank of 6. The following query asks for the hypothetical rank.

SELECT RANK(98000.00) WITHIN GROUP
       (ORDER BY SUM(f.purchase_price)) as hrank,
FROM purchases f, product p
WHERE f.product_id = p.product_id and p.category = 'HDRW'
GROUP BY p.product_id;

HRANK
----------
         6

The way to recognize a hypothetical rank function in a query is the WITHIN GROUP clause and a constant expression within the RANK() function.

3.7.3 User-defined aggregates

Oracle 9i introduced a feature to allow users to define custom aggregate functions. These aggregates can be used like regular aggregates or like analytical functions as reporting aggregates and window aggregates. Userdefined aggregates can be used when there is a complex function that cannot be expressed in SQL or a complex data representation such as objects or LOBs. These are typically useful in financial applications.

User-defined aggregates are part of Oracle's extensibility framework. To define a user-defined aggregate, you must first define a type that implements the ODCIAggregate interface. You then declare a function that uses this type to perform aggregation. The implementation of the aggregate functions can be in any procedural language such as C, PL/SQL, or Java.

The ODCIAggregate interface consists of the following functions:

  • ODCIAggregateInitialize() initializes the aggregate value at the start of processing.

  • ODCIAggregateIterate() updates the aggregate for a new row of data.

  • ODCIAggregateTerminate() returns the aggregate value and ends processing.

  • ODCIAggregateMerge() is used to support parallel computation of the aggregation. The aggregate is computed on different pieces of the data and finally the ODCIAggregateMerge() is called to combine the results. The PARALLEL_ENABLE clause must be specified on the aggregate function to enable this.

For instance, suppose you have a proprietary sales forecasting algorithm that takes the sales numbers and comes up with an estimate for sales for the next year. You can define a user-defined aggregate for this as follows. The SalesForecastFunction type implements the ODCIAggregate interface. (We omit the implementation here for lack of space.) The function SalesForecast() is declared as an aggregate function using the SalesForecastFunction.

CREATE OR REPLACE TYPE SalesForecastFunction AS OBJECT (
  data number,
  STATIC FUNCTION ODCIAggregateInitialize
                  (ctx IN OUT SalesForecastFunction)
      RETURN number,
  MEMBER FUNCTION ODCIAggregateIterate
                  (self IN OUT SalesForecastFunction,
                   value IN number) RETURN number,
  MEMBER FUNCTION ODCIAggregateTerminate
                  (self IN OUT SalesForecastFunction,
                   returnValue OUT number,
                   flags IN number) RETURN number,
  MEMBER FUNCTION ODCIAggregateMerge
                  (self IN OUT SalesForecastFunction,
                   ctx2 IN OUT SalesForecastFunction)
                  RETURN number
);
/

CREATE OR REPLACE TYPE BODY SalesForecastFunction
IS
...
END;
/

CREATE or REPLACE FUNCTION SalesForecast(x number) RETURN number
PARALLEL_ENABLE AGGREGATE USING SalesForecastFunction;

This function can then be used in a SQL query in place of any aggregate, as follows. You can also use the DISTINCT flag to remove a duplicate column value prior to aggregation.

SELECT p.category, SUM(f.purchase_price) sales,
       SalesForecast(f.purchase_price) as salesforecast
FROM purchases f, product p
WHERE f.product_id = p.product_id
GROUP BY p.category;

CATE      SALES SALESFORECAST
---- ---------- -------------
ELEC 33327213.9    36659935.3
HDRW 2962332.61    3258565.87
MUSC 3223204.06    3545524.47
Hint:

Before you implement a user-defined aggregate, check if your aggregate can be handled by existing SQL aggregates, since they would give better performance. The CASE function discussed next can be used to handle a wide variety of complex computations.

3.7.4 CASE expression

The CASE expression allows you to return different expressions based on various conditions. The simple CASE statement is identical to a DECODE statement, where you can return different values depending on the value of an expression. The searched CASE statement allows you more flexibility, as illustrated by this example. Suppose you had to target a new advertisement and wanted to bucket all your products based on the total sales they contribute. You could use a CASE expression, as follows:

SELECT f.product_id, SUM(f.purchase_price) as sales,
      CASE WHEN SUM(f.purchase_price) > 150000 THEN 'High'
           WHEN SUM(f.purchase_price)
                BETWEEN 100000 and 150000 THEN 'Medium'
           WHEN SUM(f.purchase_price)
                BETWEEN 50000 and 100000 THEN 'Low'
           ELSE 'Other' END as sales_value
FROM purchases f
GROUP BY f.product_id;

PRODUCT_      SALES  SALES_VALUE
-------- ---------- ------------
SP1000    147702.64       Medium
SP1001    106161.88       Medium
SP1010    122422.72       Medium
SP1011     159622.6       High
SP1012    143332.88       Medium
SP1013      89592.8       Low
SP1014    129582.53       Medium
SP1015    110505.91       Medium
SP1016    161189.64       High
SP1017     93159.68       Low
SP1018     174191.4       High
...

You can use the CASE expression anywhere you use a column or expression, including inside aggregate functions. The combination of aggregates and CASE expressions can be used to compute complex aggregations. For instance, suppose you want to see your average sales if you ignore purchases less than $10, add a 10 percent luxury charge on all purchases over $500, and close down shop in Massachusetts; you could use the following statement:

SELECT AVG(f.purchase_price) as current_avg_sales,
       AVG (CASE WHEN f.purchase_price < 10 THEN 0
                 WHEN f.purchase_price > 500
                      THEN 1.1 * f.purchase_price
                 WHEN f.state_id = 'MA' THEN 0
                 ELSE f.purchase_price END) projected_avg_sales
FROM purchases f;

CURRENT_AVG_SALES PROJECTED_AVG_SALES
----------------- -------------------
       272.493792          293.053466

3.7.5 WITH clause

Analytical applications often involve queries that contain complex subqueries. In fact, the same subquery may appear multiple times in the query. Oracle 9i introduced the WITH clause, which allows you to name a subquery and then subsequently use the name instead of that subquery within a statement. If the same subquery appears multiple times in a query, Oracle will automatically materialize that subquery into a temporary table and reuse it when executing the query. The temporary table will live only for the duration of the query and will be automatically deleted when the execution is complete.

The WITH clause can improve the readability of complex queries and also improve performance for queries needing repeated computation.

The following example shows how a WITH clause can be used to simplify a query. Let us say, we are finding the months for which the sales were the highest, using the following query:

SELECT s.category, s.month, s.monthly_prod_sales
FROM (SELECT p.category, t.month,
             SUM(f.purchase_price) as monthly_prod_sales
      FROM product p, purchases f, time t
      WHERE f.product_id = p.product_id
        AND f.time_key = t.time_key
      GROUP BY p.category, t.month) s
WHERE s.monthly_prod_sales
  IN (SELECT MAX(v.monthly_sales)
      FROM (SELECT p.category, t.month,
                  SUM(f.purchase_price) as monthly_sales
            FROM product p, purchases f, time t
            WHERE f.product_id = p.product_id
              AND f.time_key = t.time_key
            GROUP BY p.category, t.month) v
      GROUP BY v.month);

We can see that the subqueries with alias "s" and "v" are identical, so let us rewrite this statement to use the WITH clause. We first pull out the common subquery and give it a name, say product_sales_by_month. Then, wherever we used this subquery, we will use this name, resulting in the following statement:

WITH product_sales_by_month <- name the subquery
AS
(
 SELECT p.category, t.month,
        SUM(f.purchase_price) as monthly_prod_sales
 FROM product p, purchases f, time t
 WHERE f.product_id = p.product_id
   AND f.time_key = t.time_key
 GROUP BY p.category, t.month
)
SELECT s.category, s.month, s.monthly_prod_sales
FROM   product_sales_by_month s              <- use name here
WHERE  s.monthly_prod_sales
   IN (SELECT MAX(v.monthly_prod_sales)
       FROM product_sales_by_month v         <- use name here
       GROUP BY v.month);

This makes the query execute more efficiently, since Oracle can choose to materialize the result of the subquery into a temporary table and reuse it in both places, thereby saving repeated computation. Also the query is now much easier to read. Note that this particular query also could have been done efficiently using reporting aggregates discussed previously.

In this section, we have discussed several SQL functions in Oracle that can express complex analytical queries. All of the features we have discussed so far in this chapter can greatly benefit from parallel execution, which is discussed in the next section.


Parallelism

3.8 Parallelism

Many operations in a data warehouse involve processing large amounts of data. Bulk loads, creating indexes and summaries, sorting, and joining data from multiple tables can all take a considerable amount of time. Parallel execution can be used to reduce the time it takes to execute these operations.

Parallelism allows certain SQL statements to be divided transparently into several concurrently executing operations. By dividing the work among several processes on different processors, the statement can be completed faster than with only a single process.

In Oracle, parallel execution is performed using a parallel execution coordinator process and a pool of parallel execution servers. The Oracle process that handles the user's query becomes the coordinator process for that query. The coordinator process partitions the work to be done among the required number of parallel execution servers. It ensures that the load is balanced among the processes and redistributes work to any process that may have finished before the others. The coordinator receives the results from the parallel execution servers and assembles them into the final result.

The type of query whose performance improves the most by executing it in parallel is one that processes large amounts of data and returns a small result, such as an aggregation. For example, obtaining the total sales for the first four months of the year is shown in Figure 3.10.

Click To expand
Figure 3.10: Parallel query.

In Oracle 9i, many operations can benefit from parallelism, including DDL statements (such as CREATE TABLE, CREATE INDEX, and CREATE MATERIALIZED VIEW), partition maintenance operations (such as TRUNCATE, EXCHANGE, MERGE, and SPLIT PARTITION), SELECT statements, and DML statements. Various operations such as joins, aggregation, sorting, and DISTINCT within a SELECT statement, can be parallelized. Partition pruning and partition-wise join, discussed earlier, greatly improve performance when used with parallel execution.

With Oracle 9i Release 2, parallel DML is now possible on partitioned as well as nonpartitioned tables. In prior releases, all DML types were supported for partitioned tables; however, only parallel INSERT was supported on nonpartitioned tables.

You must use the ALTER SESSION ENABLE PARALLEL DML in the session to allow DML statements to be parallelized. PL/SQL functions and table functions can also be parallelized in some cases.

The operation to be executed in parallel can be broken up in various ways. If a table or index is partitioned, as in the previous example, the coordinator may assign each parallel execution server to work on one or more partitions. Alternatively, a table or index may be divided up into ranges of physical data blocks on disk. Each parallel execution server works in parallel on different ranges of blocks.

You can set the degree of parallelism for a table or an index using the PARALLEL clause on CREATE or ALTER TABLE, for a SQL statement using optimizer hints and for a session using the ALTER SESSION FORCE PARALLEL statement. The following statement sets the degree of parallelism to four for the purchases table.

ALTER TABLE purchases PARALLEL 4;

If not specified using these methods, Oracle will determine the degree of parallelism using the number of CPUs or number of partitions. For good performance, the number of partitions should be a multiple of the degree of parallelism. Otherwise, some parallel execution servers may remain idle waiting for others to complete a certain operation before beginning the next one. The query optimizer will try to use the specified degree of parallelism, but if several queries are competing for resources, it may adjust it to a lower value accordingly.

At database startup, Oracle will start as many parallel execution servers as specified by the PARALLEL_MIN_SERVERS initialization parameter. These are available for use by any parallel operation. The query coordinator obtains the required number of parallel execution servers from the pool when needed to execute a parallel operation. When processing is complete, the coordinator returns the parallel execution servers to the server pool.

If there are a large number of concurrent users executing parallel statements, additional parallel execution servers can be created. The initialization parameter, PARALLEL_MAX_SERVERS, specifies the maximum number of server processes to create. When they are no longer needed, the parallel execution servers that have been idle for a period of time are terminated. The pool is never reduced below the PARALLEL_MIN_SERVERS parameter. If parallel execution servers are unavailable, the operation will execute in serial. Sometimes a parallel operation cannot be executed efficiently unless a certain number of servers are available. The initialization parameter, PARALLEL_MIN_PERCENT, is used to specify the minimum percentage of requested parallel execution servers needed for the operation to succeed. If this percentage is not met, Oracle will return an error and you can try the operation again later.

In Oracle 9i, a new initialization parameter, PARALLEL_AUTOMATIC_TUNING, has been added to set default values for several parameters for tuning parallel execution. The new automatic memory management feature (section 3.6) eliminates the need to manually tune the work area sizes for many operations such as a hash join and sorting. This greatly eases the use of the parallel execution feature.

3.8.1 Hardware requirements for parallel execution

The data throughput requirements of a warehouse can often only be met with parallel execution. It is important to have the right hardware and system characteristics to get the benefits of parallel execution. The hardware selected for the data warehouse or data mart must be a multiprocessor machine, either a symmetric multiprocessor (SMP), cluster, or massively parallel processor (MPP) system. Uniprocessors have only a single CPU and are, therefore, appropriate only for small data marts.

MPP systems have several nodes connected together by a high-speed interconnect. In this "shared nothing" architecture, each node consists of a CPU, memory, and local disks and runs its own version of the operating system. To run on an MPP system, data must be partitioned appropriately across the disks. MPP systems can provide good performance but are not fault tolerant, and loss of a node can cause part of the data to be unavailable. They are also hard to manage, since addition of new nodes may require repartitioning of the data. Further, since all queries may not access all the data, the nodes may not be utilized uniformly.

SMP systems have multiple CPUs that share memory and disks via a common bus. There is only one version of the operating system running, which schedules tasks to execute on each CPU. Shared-disk systems require a distributed lock manager to coordinate data access by various nodes, which adds some overhead. SMP systems, however, do not suffer from the maintenance and fail-over problems of MPP systems and are a good choice for a warehouse. However, SMP systems are only available in configurations up to a certain number of processors, usually 64.

Clusters combine multiple nodes of a uniprocessor, or SMP system, together. Clusters are a good way to scale your data warehouse with increasing data needs. Clusters also provide excellent reliability, since failure of a node does not make data unavailable. Oracle 9i introduced Real Application Clusters (RAC) technology, which provides improved cluster support for the Oracle database. In a clustered database system, each node of the cluster runs an instance of the Oracle database server, all accessing the same underlying data. In addition to sharing the data on disk, RAC also provides sharing of buffer caches of the instances, thereby reducing disk I/O and improving performance. If parallel execution is desired, a single SQL query can be distributed across slave processes running on multiple database instances. Oracle will determine at run time whether to run the query on a single instance or multiple instances to keep internode communication to a minimum.

Parallelism improves performance only if you have the necessary hardware resources available. If your CPUs or disks are already heavily loaded, upgrade your hardware configuration before enabling parallelism.


Summary

3.9 Summary

In this chapter, we looked at several features in the Oracle database specially suited to data warehouses. Partitioning the data improves the availability of your system and keeps it manageable. Parallel execution improves the scalability of your system as the data size grows. Oracle provides several mechanisms to improve query performance, such as bitmap indexes, star transformation, partition-wise join, and partition pruning. Complex aggregation and reporting needs can be met through the new aggregation and analytical functions in the database. Automatic memory management takes the pain out of tuning work area sizes.

A well-designed warehouse making proper use of these features can deliver excellent query performance for large amounts of data. However, this is not all-in the next chapter, we will discuss another indispensable tool for any data warehouse designer: query rewrite with summary tables.


Chapter 4: Summary Management

4.1 Summary tables

A common technique used in data warehouses is to precompute and store results of frequent queries. This is especially beneficial when the queries involve aggregation, because the result is usually much smaller than the detailed data used to produce the result. An example of such a query is a monthly sales report for the business. As multiple users are interested in the total sales of each product for each month, the data would be selected, joined, sorted, and aggregated over and over again for each user. Rather than wasting resources reexecuting the same query repeatedly, the result could be precomputed and saved in a table. Such precomputed results are often called summaries or summary tables.

Table 4.1 shows a summary containing the total number of items sold for each month of the year. Summary tables are usually much smaller than the tables containing the detail data. In this example, 17 rows of detailed sales transaction data is summarized into 10 rows. Depending on the data, the reduction in size and therefore the improvement in query performance can be quite significant.

Table 4.1: Summary Table
Detail Sales Transactions
Date | Customer | Product
1/5/02 | Smith | Book
1/6/02 | Jones | Tent
2/5/02 | Smith | Book
2/5/02 | Mills | Book
3/5/02 | Smith | Film
3/7/02 | Mills | Book
3/5/02 | Smith | Film
4/1/02 | Smith | Book
4/2/02 | Jones | Book
4/3/02 | Smith | Tent
4/3/02 | Jones | Tent
5/5/02 | Peters | Book
5/6/02 | Smith | Book
6/1/02 | Jones | Book
6/5/02 | Smith | Tent
6/6/02 | Jones | Tent
6/5/02 | David | Tent
Total Sales by Product, by Month
Month | Product | QTY Sold
1/02 | Book | 1
1/02 | Tent | 1
2/02 | Book | 2
3/02 | Book | 1
3/02 | Film | 2
4/02 | Book | 2
4/02 | Tent | 2
5/02 | Book | 2
6/02 | Book | 1
6/02 | Tent | 3

Oracle 8i, with its Summary Management feature, made a huge advance in the way people used summary tables by letting the database manage summaries transparently.

4.1.1 Why do you need Summary Management?

To understand why Summary Management is needed, let us first look at some of the tasks involved in managing summary tables.

A summary is essentially the precomputed result of a SQL query. Once a summary has been created, the result of the query can be obtained from the summary. A summary can also be used to answer other related queries. For instance, if we were interested in the total sales of each product for each year, the result can be obtained by adding the months for that year together, since months roll up into years. To use a summary, the query has to be modified somehow to reference that summary. In some situations this would mean that application SQL must be modified. Alternatively, users must be informed of the existence of summarized data and trained on which summary tables to use for each particular query. Thus, while summary tables improve query performance, managing these summaries can be quite a task.

As new detail data is loaded into the warehouse, the data in the summary is no longer synchronized with the detail tables. When this happens, the summary is said to be stale. Table 4.2 shows a stale summary. In order to bring it up-to-date with the detail data, the stale summary must be refreshed. A summary can be rebuilt when new data is loaded into the warehouse. This is known as complete refresh. In some cases, it is possible to incrementally refresh the summary with only the new or changed data. Before Summary Management with Oracle 8i , refreshing summaries involved complex custom-built procedures.

Table 4.2: Stale Summary
Detail Sales Transactions
Date | Customer | Product |
1/5/02 | Smith | Book |
1/6/02 | Jones | Tent |
2/5/02 | Smith | Book |
2/5/02 | Mills | Book |
3/5/02 | Smith | Film |
3/7/02 | Mills | Book |
3/5/02 | Smith | Film |
4/1/02 | Smith | Book |
4/2/02 | Jones | Book |
4/3/02 | Smith | Tent |
4/3/02 | Jones | Tent |
5/5/02 | Peters | Book |
5/6/02 | Smith | Book |
6/1/02 | Jones | Book |
6/5/02 | Smith | Tent |
6/6/02 | Jones | Tent |
6/5/02 | David | Tent |
7/1/02 | Smith | Book | New detail not yet in summary
7/2/02 | Jones | Book
7/1/02 | David | Tent
Total Sales by Product, by Month
Month | Product | QTY Sold
1/02 | Book | 1
1/02 | Tent | 1
2/02 | Book | 2
3/02 | Book | 1
3/02 | Film | 2
4/02 | Book | 2
4/02 | Tent | 2
5/02 | Book | 2
6/02 | Book | 1
6/02 | Tent | 3

Choosing which summaries to create requires an understanding of the workload-what types of questions users are asking, and how often the same information is being requested. The number of possible summary tables that could be created is very large. Since summaries consume disk space and take time to refresh, it is important to select few summaries that produce the most performance benefits.

In conclusion, before using summaries, the following questions must be addressed:

  1. What is the best set of summaries to create?

  2. Do users have to be aware of summaries? If so, how will the users know what summaries exist and when to use them? If summaries are later determined not to be that useful and are dropped, users also need to know about this.

  3. As the detail data change, how will the summaries be kept up-to-date?

Oracle's Summary Management has the answer to all these questions. With this feature, summaries can be created directly in the Oracle database. Mechanisms are provided to keep your summaries up-to-date with changes in underlying data. Further, Oracle will transparently rewrite your queries to use these summaries, so users do not have to be aware of the summaries.

4.1.2 Summary Management with Oracle

The Summary Management feature in Oracle includes the following components:

  • A database entity known as a Materialized View, which is essentially the summary table

  • Query Rewrite, which transparently rewrites SQL queries to use materialized views

  • A mechanism to refresh the summaries using either complete or incremental refresh

  • A Summary Advisor, which recommends what summaries to create

  • Dimensions, which provide an ability to declare hierarchical relationships such as rollups in the data, to assist query rewrite

With the Summary Advisor you can easily determine the materialized views to create for a given set of queries to fit a specified amount of space. Once the summaries have been created and enabled for query rewrite, queries will automatically use these summaries. A significant benefit of this is that the end users and database applications no longer need to be aware of the existence of the summaries. Many query tools, such as Microstrategy's DSS Agent and Information Advantage's Decision Suite, also provide some query rewrite capabilities (also known as aggregate navigation). However, unlike these tools, query rewrite in Oracle is very general and is not limited to star schemas or queries with aggregation. Any client tool can take advantage of this feature in the database server to provide aggregate navigation capabilities. For instance, Oracle Discoverer uses the Summary Management features in the Oracle database to improve query response time.

Summary Management also provides complete and fast refresh procedures, so that the summaries can be updated when new detail data is loaded into the warehouse. This eliminates the need to write complex incremental refresh programs.

We will now look at each aspect of Summary Management in detail.


Creating a materialized view

4.2 Creating a materialized view

In Oracle, a summary is created using an object known as a materialized view (MV). A materialized view precomputes and stores the result of a SQL query. Note that a summary is typically an aggregate query; however, materialized views can be created for any query. In this respect, a materialized view is quite similar to a conventional view. The difference is that by materializing the view, you save the results in the database. In the remainder of this chapter, we will use the term materialized view instead of summary.

Hint:

Materialized views can also be used for replication of data. Before Oracle 8i, such materialized views were called snapshots. Materialized views include all the features that were available with Snapshots and more. In this book, we will only focus on the application of materialized views to data warehousing.

A materialized view is created using the CREATE MATERIALIZED VIEW statement. The following example creates a materialized view named MONTHLY_SALES_MV, which contains the total sales of each product for each month.

CREATE MATERIALIZED VIEW MONTHLY_SALES_MV
PCTFREE 0 TABLESPACE mview             <- storage parameters
STORAGE (initial 64k
   next 64k pctincrease 0)
BUILD IMMEDIATE                        <- when to populate it
REFRESH FORCE                          <- how to refresh it
ON DEMAND                              <- when to refresh it
ENABLE QUERY REWRITE                   <- use query rewrite or not
AS
SELECT t.month, t.year, p.product_id,  <- what it contains
       SUM (f.purchase_price) as sum_of_sales,
       COUNT (f.purchase_price) as total_sales,
       COUNT(*) as cstar
FROM time t, product p, purchases f
WHERE t.time_key = f.purchase_date AND
      f.product_id = p.product_id
GROUP BY t.month, t.year, p.product_id;

The materialized view definition contains the following parts, highlighted with arrows in the example:

  • A description of how to physically store the materialized view (i.e., the storage clause)

  • When to populate it-immediately upon creation or later-or, is it an existing summary table that must be converted into a materialized view

  • How to refresh it when data in the underlying detail tables has changed

  • When to refresh it-at the end of each transaction or when explicitly requested

  • Whether to use it for query rewrite or not

  • A SELECT statement, which describes the contents of the materialized view

Figure 4.1 shows a materialized view being created in Oracle Enterprise Manager. All the options for the CREATE MATERIALIZED VIEW statement may be specified via this graphical interface. There is also an option to collect statistics on the materialized view upon creation. The Explain button can be used to run the EXPLAIN_MVIEW utility, which helps identify the capabilities of the materialized view. See section 4.4 for a discussion of EXPLAIN_MVIEW.

Click To expand
Figure 4.1: Creating a materialized view in Oracle Enterprise Manager.

We will now explain each clause of the materialized view creation statement in detail.

4.2.1 Physical storage for the materialized view

The materialized view has a storage specification, where you can specify the tablespace to store the data in, the initial allocation size, and the size of its extents.

In our example, the materialized view is being placed in the tablespace called "MVIEW," the first extent will be 64 K, and all subsequent extents will be 64 K.

4.2.2 When should the materialized view be populated with data?

The materialized view definition describes when you would like the materialized view to be populated with data. If you specify BUILD IMMEDIATE, as in the example code segment, the materialized view is populated immediately upon creation. If you specify BUILD DEFERRED, the materialized view will be populated when you perform the REFRESH operation. If you have an existing summary table that you would like to manage using Oracle's Summary Management, you can use the ON PREBUILT TABLE clause. This converts the existing table into a materialized view and makes it available to query rewrite, if desired.

Hint:

A materialized view is not considered by query rewrite until it has been populated with data.

4.2.3 How should the materialized view be refreshed?

As new detail data is periodically loaded into the data warehouse, the materialized views have to be refreshed to reflect the changes. Four refresh options are available:

  • Complete

  • Fast

  • Force

  • Never

A materialized view can be completely rebuilt by specifying REFRESH COMPLETE. Or, it can be incrementally updated by specifying the REFRESH FAST option. It is usually faster to perform a fast refresh than a complete refresh; however, it is not always possible to do a fast refresh.

The example shown previously, uses REFRESH FORCE, which means Oracle will perform a fast refresh if possible and only do a COMPLETE refresh if necessary. Materialized views that use the NEVER REFRESH option will never be refreshed by any of the procedures supplied by Oracle. This option can be useful if you have legacy refresh procedures that cannot be eliminated or would like to store some historical data that must not be updated.

4.2.4 When should the materialized view be refreshed?

As the underlying detail data change, the materialized views that are based on those detail tables become stale and no longer reflect the results of summarizing all the detail data. The most common ways of refresh are ON COMMIT or ON DEMAND.

If your business is such that it requires the materialized view be kept up-to-date with the detail data at the transaction level, the materialized view can be refreshed at the end of each transaction, by specifying the ON COMMIT option.

On the other hand, with ON DEMAND refresh you must manually request that the materialized view be refreshed. ON DEMAND refresh allows you to control when the materialized view will be refreshed. This is useful in a warehouse where new data is loaded in a batch, after which the materialized views must be refreshed.

You can also specify that a materialized view be refreshed on a periodic schedule by specifying a start date and subsequent refresh intervals using the START WITH and NEXT clauses.

The facilities for refreshing materialized views will be discussed in section 4.3.

4.2.5 Enabling the materialized view for query rewrite

In most cases you will want queries to be transparently rewritten to use your materialized views. Whenever a materialized view is defined, if you wish it to be eligible for query rewrite, then include the clause ENABLE QUERY REWRITE, as shown in the example. If you don't want a materialized view to be used for query rewrite, use the DISABLE QUERY REWRITE clause. If you already have materialized views and would like Oracle to refresh them, but not make them eligible for query rewrite, disable query rewrite in the materialized view definition.

The details of using query rewrite will be discussed in section 4.6.

4.2.6 Specifying the contents of the materialized view

The materialized view definition includes a SELECT statement, which describes its contents. A typical query for a materialized view could include any of the following:

  • A WHERE clause, which joins the fact table and one or more dimension tables. In our example, the fact table, PURCHASES, is joined with the dimension tables, TIME and PRODUCTS. Selections may also be present to restrict the data in the materialized views.

  • One or more aggregate operators. In our example, COUNT is used to obtain the total sales. Operators that can be included are SUM, MIN, MAX, AVG, COUNT(*), COUNT, COUNT(DISTINCT x), VARIANCE, and STDDEV. Analytical functions and window aggregates discussed in Chapter 3 can also be used in a materialized view.

  • A GROUP BY clause. In the example, we are counting the total items sold by the columns year, month, and product_id.

The materialized view in the previous example includes aggregation and hence is called a materialized aggregate view.

A materialized view can also be used to join two or more tables without any aggregation. This type of a materialized view is called a materialized join view (MJV) and is used to precompute expensive joins. A materialized join view can be used in lieu of a bitmap join index. A bitmap join index is geared more toward star queries and will work better than an MJV if you want to compute combinations of selections against the index keys. On the other hand, a materialized join view can be used to answer a wider class of queries using query rewrite. However, an MJV will typically be much larger than a bitmap join index.

The following example, shows a materialized join view that is used to compute the join between PURCHASES and CUSTOMER tables.

CREATE MATERIALIZED VIEW customer_purchases_mv
BUILD IMMEDIATE
REFRESH COMPLETE
AS
SELECT c.sex, c.occupation, f.purchase_price
FROM purchases f,  customer c
WHERE f.customer_id = c.customer_id

The examples in this chapter will focus on these two types of materialized views.

4.2.7 Using summary management with existing summary tables

If you already have a data warehouse with summaries stored in user tables and do not want to completely recreate them, Summary Management can still be used to manage them and to perform query rewrite.

To register your summary table with Summary Management, create a materialized view using the ON PREBUILT TABLE clause. This converts the existing table into a materialized view. If you specify the ENABLE QUERY REWRITE clause, the materialized view can be used for query rewrite. For example, you may already have a table named monthly_customer_sales, which summarizes the amount of money each customer spent each month, as represented by the following query:

SELECT t.year, t.month, c.customer_id,
       SUM(f.purchase_price) as dollar_sales
FROM time t, purchases f, customer c
WHERE f.time_key = t.time_key AND
      f.customer_id = c.customer_id
GROUP BY t.year, t.month, c.customer_id;

The monthly_customer_sales table is described as follows:

SQL> describe monthly_customer_sales;

Name                                      Null?    Type
----------------------------------------- -------- ------------
YEAR                                               NUMBER(4)
MONTH                                              NUMBER(2)
CUSTOMER_ID                                        VARCHAR2(10)
DOLLAR_SALES                                       NUMBER

If you'd like to use this table as a materialized view and make it eligible for query rewrite, you can issue the following statement. Note that when creating the materialized view using an existing table, the materialized view must have the same name as the table.

CREATE MATERIALIZED VIEW monthly_customer_sales
ON PREBUILT TABLE
ENABLE QUERY REWRITE
AS
SELECT t.year, t.month, c.customer_id,
       SUM(f.purchase_price) AS dollar_sales
FROM time t, purchases f, customer c
WHERE f.time_key = t.time_key AND
      f.customer_id = c.customer_id
GROUP BY t.year, t.month, c.customer_id;

This materialized view can now be used to transparently rewrite queries and can be refreshed using Oracle's supplied refresh procedures.

A word of caution when creating materialized views with the PRE-BUILT clause-Oracle will not verify that the existing table corresponds to the query you use when defining the materialized view. You must ensure that the contents of the existing table correctly represent the summarization of the detail data. Otherwise, you may see incorrect results with query rewrite.

Hint:

For query rewrite to use a materialized view with the PREBUILT option, you must set the initialization parameter QUERY_REWRITE_INTEGRITY to TRUSTED or STALE_TOLERATED, as explained in section 4.6.8.

4.2.8 Partitioning the materialized view

You can partition a materialized view as you would a table, using any of the partitioning methods available in Oracle. In a warehouse, it is common to partition materialized views in the same way as the fact table. Range-list partitioning may also be a good way to partition a materialized view when it contains multiple dimensions such as time (range) and geography (list).

In the following example, the materialized view is partitioned using range partitioning.

CREATE MATERIALIZED VIEW MONTHLY_SALES_MV
PARTITION by RANGE (time_key)
 (  partition purchases_jan2002
     values less than (TO_DATE('01-02-2002', 'DD-MM-YYYY'))
     pctfree 0 pctused 99
     storage (initial 64k next 64k pctincrease 0)
     tablespace purchases_jan2002 ,
    partition purchases_feb2002
     values less than (TO_DATE('01-03-2002', 'DD-MM-YYYY'))
     pctfree 0 pctused 99
     storage (initial 64k next 64k pctincrease 0)
     tablespace purchases_feb2002 ,
    partition purchases_mar2002
     values less than (TO_DATE('01-04-2002', 'DD-MM-YYYY'))
     pctfree 0 pctused 99
     storage (initial 64k next 64k pctincrease 0)
      tablespace purchases_mar2002
)
BUILD IMMEDIATE
REFRESH FORCE
ON DEMAND
ENABLE QUERY REWRITE
AS
SELECT t.time_key, p.product_id, SUM (f.purchase_price) as
sum_of_sales
FROM time t, product p, purchases f
WHERE t.time_key = f.purchase_date AND
      f.product_id = p.product_id
GROUP BY t.time_key, p.product_id;

Partitioning a materialized view can speed up refresh by using parallel DML. It also enables queries that use the materialized view to take advantage of optimizations such as partition pruning.

4.2.9 Indexing the materialized view

You can also build indexes on your materialized view to improve the performance of your queries. The techniques used to index materialized views are similar to those for any table.

In the following example, a concatenated index is created on the grouping columns of the MONTHLY_SALES_MV materialized view, shown earlier in this chapter. Also, bitmap indexes are created on the grouping columns. These indexes will improve performance of queries that get rewritten to use the materialized view.

CREATE INDEX easydw.products_by_month_concat_index
ON MONTHLY_SALES_MV  (month, year, product_id)
pctfree 5
tablespace indx
storage (initial 64k next 64k pctincrease 0) ;

CREATE BITMAP INDEX easydw.total_products_by_month_index
ON monthly_sales_mv (month, year);

CREATE BITMAP INDEX easydw.total_products_by_id_index
ON monthly_sales_mv (product_id);

Oracle automatically creates an index on some materialized views to speed up fast refresh. To suppress creation of this index, use the NO INDEX clause in the CREATE MATERIALIZED VIEW statement.

4.2.10 Security of materialized views

Some information in the data warehouse may have restricted access, and it is important to ensure that the appropriate security policies are implemented regarding access to materialized views. You may want to allow users access to a materialized view but not allow them to see the underlying detail data. For example, you may allow a user to see the average salary by department but not to see an individual employee's salary. On the other hand, you want to ensure that materialized views and query rewrite are not used as a mechanism for bypassing security. As with all database objects, Oracle provides a privilege model for creating and altering materialized views.

To create a materialized view in a user's own schema, the user must have the CREATE MATERIALIZED VIEW privilege. To create a materialized view in another user's schema, the creator must have the CREATE ANY MATERIALIZED VIEW privilege. The owner of the schema where the materialized view will be placed must have the CREATE TABLE and CREATE INDEX privilege and must be able to execute the materialized view's defining query.

Hint:

It is helpful to remember that most privilege checks for a materialized view are applied to the owner of the schema in which the materialized view is placed. The creator only needs the CREATE/ALTER (or CREATE/ALTER ANY) MATERIALIZED VIEW privilege. Also, the privilege checks are applied during creation and altering and not during query rewrite.

To enable a materialized view for query rewrite, the owner of the materialized view must have the QUERY REWRITE object privilege on any tables (referenced in the materialized view) that are outside the owner's schema. Alternatively, the owner must have the GLOBAL QUERY REWRITE system privilege. (Note that the QUERY REWRITE system privilege in Oracle 8i is now deprecated.)

To refresh a materialized view in another schema, the user issuing the refresh procedure must have the ALTER ANY MATERIALIZED VIEW privilege.

To create a materialized view with REFRESH ON COMMIT option, the owner must have the ON COMMIT object privileges on all tables (referenced in the materialized view) outside the schema or have the ON COMMIT system privilege.

In the following example, the user EASYDW is granted the following privileges to create materialized views and dimensions and to allow these to be used for query rewrite.

-- Add privileges
GRANT select any table to easydw;
GRANT execute any procedure to easydw;

-- Add privileges for summary management
GRANT create materialized view to easydw;
GRANT drop materialized view to easydw;
GRANT alter materialized view to easydw;
GRANT create any dimension to easydw;
GRANT alter any dimension to easydw;
GRANT drop any dimension to easydw;
GRANT global query rewrite to easydw;


Refresh

4.3 Refresh

As new data is loaded into the warehouse, any materialized view based on this data must be updated. This operation is known as refresh. The frequency of refresh determines how recent the summary data is with respect to the detail data. The refresh operation can be very time consuming depending on the amount of data involved. You must carefully determine your refresh policy based on available resources and your application's need for current data. If your application needs the materialized view to be synchronized with the detail data at all times, choose the ON COMMIT option. With this option, Oracle will automatically refresh the materialized view when the changes to detail data get committed. If your warehouse gets new data only once a day during a specified maintenance window, deferring the refresh until then would be the right choice. Deferred or ON DEMAND refresh can be done using the DBMS_MVIEW package.

The refresh policy must be chosen when creating the materialized view. You can alter it at a later time by issuing an ALTER MATERIALIZED VIEW statement.

4.3.1 Using the DBMS_MVIEW refresh procedures

Oracle provides several procedures for ON DEMAND refresh in the DBMS_MVIEW supplied PL/SQL package, as follows:

  • DBMS_MVIEW.REFRESH

  • DBMS_MVIEW.REFRESH_DEPENDENT

  • DBMS_MVIEW.REFRESH_ALL_MVIEWS

If you want to refresh all your materialized views, which is often done after bulk loading new detail data, you can use the DBMS_MVIEW.REFRESH_ALL_MVIEWS procedure, as shown in the following example. In this example, the :failures bind variable returns the number of failed refreshes.

EXECUTE DBMS_MVIEW.REFRESH_ALL_MVIEWS(:failures);

If you have materialized views that are refreshed at different times—for example, some weekly and others monthly—you can specify a list of materialized views to refresh using the DBMS_MVIEW.REFRESH procedure, as follows:

EXECUTE DBMS_MVIEW.REFRESH('monthly_sales_mv, sales_by_city');

Recall that at the time of creating the materialized view, you specify a default refresh method of COMPLETE, FAST, or FORCE. You may override this refresh method when issuing the refresh command as C (complete), F (fast), or ? (force). If you don't specify it, the refresh method given at the time of creating the materialized view will be used. In the following example, the materialized view MONTHLY_SALES_MV defined earlier as REFRESH FORCE is being refreshed using COMPLETE refresh.

EXECUTE DBMS_MVIEW.REFRESH('monthly_sales_mv', 'C');

If you want to refresh all materialized views that are based on a particular detail table that has changed, use the DBMS_MVIEW.REFRESH_DEPENDENT procedure, specifying the detail table. For example, to refresh FORCE all materialized views that have changed when the customer table is updated, you would issue the following statement.

EXECUTE DBMS_MVIEW.REFRESH_DEPENDENT(:failures, 'customer', '?')

Refresh of several materialized views can be done simultaneously using job queues. To do this, the initialization parameter JOB_QUEUE_PROCESSES must be set to a nonzero value and the atomic_refresh parameter in the refresh procedure must be set to FALSE, as follows:

ALTER SYSTEM SET job_queue_processes = 2;

EXECUTE DBMS_MVIEW.REFRESH('sales_by_month, sales_by_city', 'C',
                           atomic_refresh=>FALSE);

Alternatively, you can use Oracle Enterprise Manager to refresh your summaries. To look at the properties of each materialized view in the system, expand the Databases node in the Navigator pane, followed by the Warehouse node, then Summary Management, and finally Materialized Views (as shown in Figure 4.2 ).

Click To expand
Figure 4.2: Refreshing a materialized view in Oracle Enterprise Manager.

You can check if the materialized view is STALE by looking at the Refresh State in the Status section. If the materialized view is STALE, you can refresh it by pressing the Refresh button.

Hint:

To get an accurate picture of the refresh state, the compile state field must say VALID. If it says NEEDS_COMPILE, press the Compile button to validate the materialized view first.

Pressing the Refresh button will bring up a refresh options screen (as shown in Figure 4.3) from which you can choose the refresh type, issue the refresh and optionally update the statistics on the materialized view.

Click To expand
Figure 4.3: Refreshing a materialized view in Oracle Enterprise Manager—Options.

Once the refresh has completed, you can once again check the status of the materialized view in the Status section.

4.3.2 Fast refresh

As the volume of data in a warehouse increases, rebuilding the materialized view after each new data load can get prohibitive. Oracle provides the capability to refresh materialized views without a complete rebuild. This is known as fast refresh. The fast refresh algorithms used by Oracle either incrementally apply changes from the detail data or selectively recompute only affected portions of the materialized view. Oracle 8i supported fast refresh for materialized aggregate views after bulk loads to the data. Refresh after DML was only supported for a restricted class of materialized views. In Oracle 9i, fast refresh can handle regular DML (insert, update, and delete statements) and also partition maintenance operations.

For a materialized view to be fast refreshable after DML to the detail data, a materialized view log must be created on each of its tables. The materialized view log captures the changes done to the detail data tables by insert, update, or delete statements. Note that changes done by a bulk load such as INSERT /*+ APPEND */ or using SQL*Loader are not recorded in the materialized view log but are kept in an internal log.

In order for a materialized view to be fast refreshable, it must adhere to certain rules. The materialized view's SELECT query must include some supporting aggregates, depending on the aggregate. For each aggregate such as SUM(x) or AVG(x), you must have COUNT(x). The materialized view must also include the COUNT(*) aggregate. If you have STDDEV(x), you must have SUM(x), COUNT(x), and SUM(x * x).

The following example shows a fast refreshable summary along with the materialized view logs on its detail tables. Note that various options such as ROWID, SEQUENCE, and INCLUDING NEW VALUES must be specified for the materialized view log. The ROWID option logs the rowids of rows changed by DML statements. The INCLUDING NEW VALUES clause logs both the old and new version of a row changed by an update statement. The SEQUENCE option is new in Oracle 9i and is required to support fast refresh. If you omit this option, you will get fast refresh only under specific conditions (i.e., either when only one detail table has been changed or when only inserts have been performed). All detail table columns referenced in the materialized view's query must be included in the materialized view log.

Hint:

You must create the materialized view logs before creating the materialized views; otherwise, you will have to completely refresh the materialized views once before fast refresh can be performed.

CREATE MATERIALIZED VIEW LOG on time
WITH ROWID, SEQUENCE (time_key, month, year)
INCLUDING NEW VALUES;

CREATE MATERIALIZED VIEW LOG on purchases
   WITH ROWID, SEQUENCE (purchase_date, product_id, purchase_price)
   INCLUDING NEW VALUES;

CREATE MATERIALIZED VIEW LOG on product
   WITH ROWID, SEQUENCE (product_id)
INCLUDING NEW VALUES;

CREATE MATERIALIZED VIEW monthly_sales_mv
 PCTFREE 0 TABLESPACE summary
 STORAGE (initial 64k next 64k pctincrease 0)
 BUILD IMMEDIATE
 REFRESH FAST
 ON DEMAND
 ENABLE QUERY REWRITE
 AS
 SELECT t.month, t.year, p.product_id,
        SUM (ps.purchase_price) as sum_of_sales,
        COUNT (ps.purchase_price) as total_sales, COUNT(*)
 FROM time t, product p, purchases ps
 WHERE t.time_key = ps.purchase_date AND
       ps.product_id = p.product_id
 GROUP BY t.month, t.year, p.product_id;

You can use the EXPLAIN_MVIEW procedure (section 4.4) to determine if a given select statement can be used to create a fast refreshable materialized view.

The following example code shows a materialized join view that is fast refreshable with the ON COMMIT option. To make this type of a materialized view fast refreshable, you must include a ROWID column for each table in the materialized view. The materialized view log only needs to have the ROWID option. (In this example, we are assuming that the materialized view logs for product and purchases created earlier for the MONTHLY_SALES_MV are around.)

CREATE MATERIALIZED VIEW LOG on customer WITH ROWID;

CREATE MATERIALIZED VIEW product_customer
BUILD IMMEDIATE
REFRESH FAST ON COMMIT
AS
SELECT c.rowid r1, c.sex, p.rowid r2, p.product_id,
       f.rowid r3, f.purchase_price
FROM purchases f, product p, customer c
WHERE f.customer_id = c.customer_id
      AND f.product_id  = p.product_id;
Hint:

The same materialized view log on a given table is used for all materialized views that reference that table; hence, be sure to specify options to satisfy all the materialized views.

During fast refresh, Oracle will identify the changes that have occurred since the last refresh using the materialized view log and apply them to the materialized view. Once all materialized views have been refreshed to include the changes, the relevant rows will be purged from the materialized view log. You must never modify the materialized view log manually, as you will be forced to completely rebuild the materialized view.

In order to get good refresh performance, it is important to have accurate statistics on the data. The package DBMS_STATS should be used to gather optimizer statistics on the detail tables after each data load and prior to creating or refreshing the materialized views. It is also important to gather statistics on the materialized views upon creation and after refreshing them with large amounts of data. This will ensure that the optimizer has accurate statistics during refresh and when optimizing queries that are rewritten to use materialized views.

4.3.3 Fast refresh after partition maintenance operations

Partitioned tables are a common feature of a warehouse. Partition maintenance operations are used to reorganize the partitioned data. These include MERGE or SPLIT of existing partitions, TRUNCATE or DROP of an old partition, or EXCHANGE of a partition with a table. The use of partition maintenance operations in a warehouse is discussed in Chapter 7.

Oracle 9i introduced a new feature known as Partition Change Tracking (PCT). This allows materialized views on partitioned tables to be fast refreshed after partition maintenance operations.

When a partition maintenance operation is performed, Oracle keeps track of the updated partitions. It can then identify the data in the materialized view that corresponds to the updated partitions and recompute only that portion of the materialized view. This is referred to as PCT refresh.

Figure 4.4 shows a conceptual picture of how data changes are tracked in partition change tracking. In this example, the purchases table is partitioned by month into Jan 2002, Feb 2002 and Mar 2002 partitions. The materialized view MONTHLY_SALES_MV includes the sales data for each month. The correspondence between the detail data and data in the materialized view is shown by the dotted lines. If the Feb 2002 partition is updated, the corresponding data in the materialized view is stale and no longer available for query rewrite. Similarly, when the Apr 2002 partition is added to the purchases table, the materialized view does not contain this partition and hence cannot be used for queries for the Apr 2002 data.

Click To expand
Figure 4.4: Partition change tracking.

To enable a materialized view for partition change tracking, you must include the partitioning key of the partitioned detail tables in the materialized view. If this is not possible, you must include a special column known as the partition marker using the DBMS_MVIEW.PMARKER() function for each partitioned table. This allows Oracle to map detail table partitions to rows in the materialized view. To take advantage of PCT, the materialized view itself does not have be partitioned at all!

The following example illustrates the use of a partition key in a materialized view. In the EASYDW schema, the purchases table is partitioned by time_key. The materialized view, MONTHLY_SALES_PARTKEY_MV, has the time_key column in the SELECT list and hence is enabled for partition change tracking if the purchases table gets updated.

CREATE MATERIALIZED VIEW monthly_sales_partkey_mv
REFRESH FAST
AS
SELECT t.month, ps.time_key,
       SUM (ps.purchase_price) as sum_of_sales,
       COUNT (ps.purchase_price) as total_sales, COUNT(*)
FROM time t, purchases ps
WHERE t.time_key = ps.time_key
GROUP BY t.month, ps.time_key;

SELECT * FROM monthly_sales_partkey_mv;

     MONTH TIME_KEY  SUM_OF_SALES TOTAL_SALES   COUNT(*)
---------- --------- ------------ ----------- ----------
         1 01-JAN-02   8989040.32       31709      31709
         1 02-JAN-02        56.02           2          2
         2 01-FEB-02   3401896.62        8902       8902
         2 02-FEB-02     13811629       31203      31203
         ...
         7 02-JUL-02        56.02           2          2
         7 05-JUL-02      2240.95          45         45
         7 07-JUL-02        28.01           1          1
         7 08-JUL-02        28.01           1          1

19 rows selected.

In this materialized view, there is a row for each value of time_key and hence it is effectively stored at a daily level and could get quite large. To mitigate this problem, you could use a partition marker instead, as shown in the next example.

CREATE MATERIALIZED VIEW monthly_sales_marker_mv
REFRESH FAST
AS
SELECT t.month, DBMS_MVIEW.PMARKER(ps.rowid) as pmark,
       SUM (ps.purchase_price) as sum_of_sales,
       COUNT (ps.purchase_price) as total_sales, COUNT(*)
FROM time t, purchases ps
WHERE t.time_key = ps.time_key
GROUP BY t.month, DBMS_MVIEW.PMARKER(ps.rowid);
SELECT * FROM monthly_sales_marker_mv;

     MONTH      PMARK SUM_OF_SALES TOTAL_SALES   COUNT(*)
---------- ---------- ------------ ----------- ----------
         1      29566   8989096.34       31711      31711
         2      29567   17213525.6       40105      40105
         3      29568    429483.97         952        952
         4      29707   4524335.65       10126      10126
         5      29708   1268317.48        3158       3158
         6      29709   7092585.46        7995       7995
         7      29843      2352.99          49         49

7 rows selected.

We see that the partition marker produces one value for all rows in the same partition. This makes the materialized view more manageable in size.

Hint:

Include the partition key if there are few distinct values in each partition. If there are many distinct values for the partition key, such as time_key, using a partition marker will keep the materialized view small.

Partition change tracking is currently only supported for detail tables partitioned by range or range-hash composite partitioning. While PCT refresh is primarily useful after partition operations, this type of refresh may also be performed after regular DML and may be quite beneficial if a lot of changes have been done to detail data. Oracle will automatically try to use PCT refresh if it cannot perform fast refresh using materialized view logs.

Hint:

To find out if PCT refresh could be used on your materialized view, use the EXPLAIN_MVIEW procedure discussed in the next section.

Partition change tracking also improves the availability of a stale materialized view to query rewrite—more on this in section 4.6.


EXPLAIN_MVIEW utility

4.4 EXPLAIN_MVIEW utility

As we have seen so far, in order to get the most benefit out of a materialized view, the materialized view must conform to some rules. For instance, you may need to create materialized view logs to make fast refresh possible or add a partition marker to allow partition change tracking. The EXPLAIN_MVIEW interface is designed to help you determine what these rules are. Before creating the materialized view, you can run its defining query through the EXPLAIN_MVIEW utility. If a capability such as fast refresh is disallowed, EXPLAIN_MVIEW will point out the offending construct or missing columns. You now have the opportunity to fix these problems before expending precious resources to create the materialized view. This facility is new in Oracle 9i. EXPLAIN_MVIEW can also tell you what features are available with your existing materialized view.

EXPLAIN_MVIEW is invoked via the PL/SQL API DBMS_MVIEW.EXPLAIN_MVIEW and comes in two flavors. You can provide an existing materialized view name to it and display its capabilities. Alternatively, you can provide a SELECT statement that defines your intended materialized view, and EXPLAIN_MVIEW will project its capabilities. The results of EXPLAIN_MVIEW are placed in the MV_CAPABILITIES_TABLE table, which must be created by running the utlxmv.sql script found in the rdbms/admin directory.

In the following example, we have deliberately introduced problems with the materialized view logs to illustrate how EXPLAIN_MVIEW can be used. We have dropped the materialized view log on product and removed the INCLUDING NEW VALUES clause from the materialized view log on time.

ALTER MATERIALIZED VIEW LOG on TIME EXCLUDING NEW VALUES;

DROP MATERIALIZED VIEW LOG on PRODUCT;

BEGIN
dbms_mview.explain_mview (
'SELECT t.month, t.year, p.product_id,
        SUM (f.purchase_price) as sum_of_sales,
        COUNT (f.purchase_price) as total_sales,
        COUNT(*) as cstar
 FROM time t, product p, purchases f
 WHERE t.time_key = f.purchase_date AND
       f.product_id = p.product_id
 GROUP BY t.month, t.year, p.product_id');
END;
/

The output of EXPLAIN_MVIEW first lists all the capabilities of a materialized view and whether each one is possible (Y) or not (N). It then presents detailed information regarding each capability and the reason, if any, why it is not possible.

SELECT capability_name, possible p, related_text table,
       msgtxt  explanation
FROM MV_CAPABILITIES_TABLE;
CAPABILITY_NAME                P  TABLE     EXPLANATION
-----------------------------  -  -----     ----------------------
PCT                            N
REFRESH_COMPLETE               Y
REFRESH_FAST                   N
REWRITE                        Y
REWRITE_FULL_TEXT_MATCH        Y
REWRITE_PARTIAL_TEXT_MATCH     Y
REWRITE_GENERAL                Y

PCT_TABLE                      N  TIME      relation is not a
                                            partitioned table

PCT_TABLE                      N  PRODUCT   relation is not a
                                            partitioned table

PCT_TABLE                      N  PURCHASES no partition key or
                                            PMARKER in select list

REFRESH_FAST_AFTER_INSERT      N  PRODUCT   the detail table does
                                            not have a
                                            materialized view log

REFRESH_FAST_AFTER_INSERT      N  TIME      mv log must have new
                                            values

REFRESH_FAST_AFTER_ONETAB_DML  N            see the reason why
                                            REFRESH_FAST_AFTER_
                                            INSERT is disabled

REFRESH_FAST_AFTER_ANY_DML     N            see the reason why
                                            REFRESH_FAST_AFTER_
                                            ONETAB_DML is disabled

REFRESH_FAST_PCT               N            PCT is not possible on
                                            any of the detail
                                            tables in the
                                            materialized view

REWRITE_PCT                    N            PCT is not possible on
                                            any of the detail
                                            tables in the
                                            materialized view

From this output we can see that the PCT_TABLE capability for PURCHASES table is not possible, because that table is partitioned but its partition key (or partition marker) was not included in the materialized view. The TIME table is not partitioned and hence PCT refresh is not possible if it is updated.

Fast refresh may be possible in some situations and not others. Hence, the fast refresh capabilities are presented at three levels, so if the first one is not possible, the remaining ones are also not possible and so on:

  • REFRESH_AFTER_INSERT (only inserts were done to the tables)

  • REFRESH_AFTER_ONETABDML (only one table is modified at a time)

  • REFRESH_AFTER_ANY_DML (there is no restriction on the type of DML)

The output of EXPLAIN_MVIEW shown here indicates that the mate- rialized view log on the TIME table was missing the INCLUDING NEW VALUES clause and hence fast refresh using materialized view logs is not possible.

EXPLAIN_MVIEW also indicates if query rewrite has been enabled and if it is possible in general or only using text match mode.

Hint:

To find out the detailed reasons why a certain query did not rewrite using a certain materialized view, use EXPLAIN_REWRITE instead. It is discussed later in this chapter.

Let us now correct the problems with the materialized view logs and rerun the utility. You can see that fast refresh is now possible, as indicated by REFRESH_FAST_AFTER_INSERT Y.

CREATE MATERIALIZED VIEW LOG on product
       WITH ROWID, SEQUENCE (product_id)
INCLUDING NEW VALUES;

ALTER MATERIALIZED VIEW LOG on TIME including new values;

BEGIN
dbms_mview.explain_mview (
'SELECT t.month, t.year, p.product_id,
        SUM (f.purchase_price) as sum_of_sales,
        COUNT (f.purchase_price) as total_sales,
        COUNT(*) as cstar
 FROM time t, product p, purchases f
 WHERE t.time_key = f.purchase_date AND
       f.product_id = p.product_id
 GROUP BY t.month, t.year, p.product_id');
END;
/
SELECT capability, possible
FROM MV_CAPABILITIES_TABLE
WHERE capability_name LIKE 'REFRESH%';

CAPABILITY_                    P
------------------------------ -
REFRESH_COMPLETE               Y
REFRESH_FAST                   Y
REFRESH_FAST_AFTER_INSERT      Y
REFRESH_FAST_AFTER_ONETAB_DML  Y
REFRESH_FAST_AFTER_ANY_DML     Y
REFRESH_FAST_PCT               N

If you wanted to look at the capabilities of an existing materialized view, for example, MONTHLY_SALES_MV—you would use EXPLAIN_MVIEW, as follows:

EXECUTE DBMS_MVIEW.EXPLAIN_MVIEW('EASYDW.MONTHLY_SALES_MV');

EXPLAIN_MVIEW also has interfaces to return the results in a PL/SQL array rather than in a table.

The CREATE MATERIALIZED VIEW screen in Oracle Enterprise Manager (Figure 4.1) has an Explain button, which can be clicked to run EXPLAIN_MVIEW. This allows you to immediately identify and correct any problems with the materialized view. Figure 4.5 is the OEM screen showing the output of EXPLAIN_MVIEW.

Click To expand
Figure 4.5: EXPLAIN_MVIEW in Oracle Enterprise Manager

You can also issue EXPLAIN_MVIEW on an existing materialized view by using the Explain button shown in Figure 4.2.


Dimensions

4.5 Dimensions

With Oracle 8i, in addition to defining primary-foreign-key relationships among tables, you can also describe the relationships between the columns within one or more tables by creating a new schema object, called a dimension. Telling Oracle about such relationships in the data makes it possible for the advisor to recommend the best set of summaries to create, for the optimizer to rewrite more queries to use the summaries, and for the OLAP tools to perform rollup and drill-down operations. We will see the power of dimensions in the section 4.6, where we talk about query rewrite.

Typically, relationships between fact and dimension tables are specified using referential integrity constraints. Dimensions are used to define the following types of relationships:

  • The hierarchical relationships among the data within the dimension tables. This lets Oracle know what data can be rolled up to higher levels within the hierarchy for summarization and how to drill down to a lower level in the hierarchy to see more detail. Hierarchical relationships are described using HIERARCHY clauses in the CREATE DIMENSION statement.

  • Functional dependencies or one-to-one relationships between two columns in a dimension table. This is useful for looking up one value based on another value in the table. For example, given a date, we can determine the day of the week. Given a month number, we can determine the month name. Given a postal code, we can determine the town. Given a product_name, we know its manufacturer. One-to-one relationships are defined using the ATTRIBUTE clause in the CREATE DIMENSION statement.

4.5.1 Creating a dimension

Once the tables that contain the dimension data is created, you can create the DIMENSION object, using the CREATE DIMENSION statement. It helps to draw a bubble diagram showing the hierarchical relationships among the columns prior to defining a dimension.

Figure 4.6 shows a bubble diagram for the geography dimension. Postal codes roll up into towns, which roll up into counties, which, in turn, roll up into countries.

Figure 4.6: Bubble diagram for the geography dimension.

To convert the bubble diagram in Figure 4.6 into a dimension definition, each bubble in the diagram becomes a LEVEL in the hierarchy, as shown in the following CREATE DIMENSION statement.

CREATE DIMENSION geography_dim
LEVEL postal_code IS customer.postal_code
LEVEL town        IS customer.town
LEVEL county      IS customer.county
LEVEL country     IS customer.country
HIERARCHY loc_rollup (
          postal_code   CHILD OF
          town          CHILD OF
          county        CHILD OF
          country
         );

The relationships between the levels are described with the HIERARCHY clause. The bubbles with arrows coming out of them in the diagram are described with the CHILD OF clause. In our example, postal code (zip code) rolls up into town; therefore, postal_code is a CHILD OF town. Town rolls up into county, therefore, town is a CHILD OF county. County rolls up into country, so county is a CHILD OF country.

For any value of a child column in a hierarchy, there must be one, and only one, value of its parent column. In the customer table used in the example, postal code 02134 is in Boston, which is in Massachusetts. Since a postal code is unique to any given town, it satisfies this rule. Postal code 02134 refers to addresses in Boston only. It cannot also be used to refer to addresses in San Francisco.

4.5.2 Defining a dimension with multiple hierarchies

Sometimes the same data can be rolled up in different ways. In our sample schema, EASYDW, as in many businesses, we use both a regular calendar and a fiscal calendar. In the regular calendar, days roll up into months, which roll up into years. In the fiscal calendar, days roll up into weeks, which roll up into fiscal quarters. This can be represented by defining multiple hierarchies within the dimension.

Figure 4.7 shows the bubble diagram for the time dimension. It contains two hierarchies, describing the different ways the time_key can be rolled up. Each arrow coming out of a bubble is described as a CHILD OF clause. Since there are two arrows coming out of the time_key bubble, there are two CHILD OF statements—one in each hierarchy. From a given date, one hierarchy tells us which week and fiscal quarter this date is in and the other tells us the month and year. At the top of every bubble diagram is the special level "ALL," representing the grand total level for that dimension. All levels of a hierarchy can be rolled up to ALL. We don't explicitly define the ALL level in the CREATE DIMENSION statement.

Click To expand
Figure 4.7: Bubble diagram for the time dimension.

The following SQL segment is an example of a dimension with multiple hierarchies.

CREATE DIMENSION time_dim
 LEVEL time_key    IS time.time_key
 LEVEL month       IS time.month
 LEVEL quarter     IS time.quarter
 LEVEL year        IS time.year
 LEVEL week_number IS time.week_number
 HIERARCHY calendar_rollup (
                            time_key  CHILD OF
                            month     CHILD OF
                            Year
                            )
 HIERARCHY fiscal_rollup (
                          time_key    CHILD OF
                          week_number CHILD OF
                          quarter
                          );

Again, for each value of a child column in a hierarchy, there is only one parent value. Any given date, say March 21, 2002 falls into one, and only one fiscal week_number and in a specific quarter.

4.5.3 Defining a dimension with attributes

Dimensions can be used to define relationships between columns within the same table that are not part of the hierarchy. Often users do not know the unique identifier for a record but know the name. For example, a customer may not know the product_id, but does know the product_name of what he or she is interested in purchasing. In relational database theory these relationships are called functional dependencies, or one-to-one relationships. A functional dependency from column A to B means that for every value of A, there is only one value of B. So if you know the value of A, then you can determine the value of B.

In a dimension definition, the ATTRIBUTE clause is used to define any functional dependencies. The ATTRIBUTE clause should be used only when there is a one-to-one relationship between the two columns.

Here we see the definition of a dimension with the attribute clause. The attribute clause tells us that given a product_id, there is only one product_name and one manufacturer. However, it does not mean that given the manufacturer we can determine the product_id.

CREATE DIMENSION product_dim
 LEVEL product_id   IS product.product_id
 LEVEL category     IS product.category
 HIERARCHY merchandise_rollup (
                               product_id   CHILD OF
                               category
                               )
 ATTRIBUTE product_id DETERMINES (product_name, manufacturer);

4.5.4 Defining a dimension with normalized tables

If you have a snowflake or other normalized schema, then your hierarchy may refer to columns in multiple tables. The dimension statement in Oracle allows you to declare such relationships as well.

Figure 4.8 shows a normalized time dimension. The bubbles and bold arrows define the rollup relationships as before. The dotted rectangles represent the tables where the levels come from. The dotted arrows show how the tables join to each other. In this example, there are separate tables for time, week, month, quarter, and year. The time table joins to the week table using the predicate time.week = week.week_number and to the month table using the predicate time.month = month.month.

Click To expand
Figure 4.8: A normalized dimension.

To convert this diagram to a CREATE DIMENSION statement, you would first define a level for each of the bubbles, specifying both the table name and column name for that level. For example, the level week is week.week_number. Next you would define your hierarchy using the level names as before. Finally, you would specify any joins that must be performed when traversing up the hierarchy using the JOIN KEY clause. Thus you would get the following CREATE DIMENSION statement:

CREATE DIMENSION time_dim
 LEVEL time_key IS time.time_key
 LEVEL month    IS month.month
 LEVEL quarter  IS quarter.quarter
 LEVEL year     IS year.year
 LEVEL week     IS week.week_number
 HIERARCHY calendar_rollup (
                            time_key CHILD OF
                            month    CHILD OF
                            year
                            JOIN KEY time.month REFERENCES month
                            JOIN KEY month.year REFERENCES year
                           )
 HIERARCHY fiscal_rollup   (
                            time_key CHILD OF
                            week     CHILD OF
                            quarter
                            JOIN KEY time.week    REFERENCES week
                            JOIN KEY week.quarter REFERENCES quarter
                            );

In the fiscal_rollup hierarchy in this example, the JOIN KEY time.week column is used to join the time table to the week level.

As discussed in Chapter 2, normalized dimension tables incur the overhead of extra joins during query processing and hence must be used with care.

4.5.5 Validating a dimension

We have said that the relationships declared by the dimension object are not automatically checked by Oracle. In order to get accurate results with query rewrite, you must ensure that these relationships hold in your data. To help with this, Oracle provides a procedure known as DBMS_OLAP.VALIDATE_DIMENSION. You can run this procedure every time new data is loaded into your dimension tables to ensure data integrity.

For example, suppose we had the following customer dimension:

CREATE DIMENSION customer_dim
  LEVEL customer IS customer.customer_id
  LEVEL town     IS customer.town
  LEVEL region   IS customer.county
  HIERARCHY  customer_zone (
                            customer CHILD OF
                            town     CHILD OF
                            region
                           )
  ATTRIBUTE town DETERMINES postal_code
  ATTRIBUTE customer DETERMINES (sex, occupation);

According to this dimension, town determines the postal_code, and every town should fall in one, and only one, county. Let us check that this is indeed the case.

SELECT distinct town, county, postal_code FROM customer;

TOWN       COUNTY     POSTAL_COD
---------- ---------- ----------
Chandlers  Hants      SO53 1TZ
Eastleigh  Hants      SO53 1LD
Liverpool  Mersey     LV1 1QT
London     London     W1 1QC
Soton      Hants      SO6 1gh
Soton2     Hants      SO1 1TF
Soton3     Hants      SO1 1TF
Soton4     Hants      SO1 1TF
Soton5     Hants      SO1 1TF
Soton6     Hants      SO1 1TF
Soton7     Hants      SO1 1TF
Soton8     Hants      SO1 1TF
Soton9     Hants      SO1 1TF

Now suppose some bad data got inserted into the customer table. The postal code for London was mistyped as W1 2QC and the town of Liver-pool was misclassified in the Hants county.

INSERT INTO customer VALUES ('AB130000', 'London', 'London',
'W1 2QC', '7-JUL-74', 'UK', 'Doctor', 'F');

INSERT INTO customer VALUES ('AB130001', 'Liverpool', 'Hants',
'LV1 1QT','18-JUN-73', 'UK', 'Doctor', 'M');

COMMIT;

If we run VALIDATE_DIMENSION, it will detect that there is a discrepancy in the HIERARCHY and ATTRIBUTE relationships declared by the customer_dim dimension.

variable run_id number;

execute dbms_olap.create_id(:run_id);
execute dbms_olap.validate_dimension('CUSTOMER_DIM', 'EASYDW',
                                     FALSE, TRUE, :run_id);

The VALIDATE_DIMENSION procedure will verify the integrity of the data and place any exceptions found in the MVIEW_EXCEPTIONS table.

SELECT distinct owner, table_name, dimension_name, relationship
FROM system.MVIEW_EXCEPTIONS
WHERE runid = :run_id;

OWNER   TABLE_NAME  DIMENSION_NAME  RELATIONSHIP
------  ----------- --------------  ------------
EASYDW     CUSTOMER   CUSTOMER_DIM     ATTRIBUTE
EASYDW     CUSTOMER   CUSTOMER_DIM     CHILD OF

The bad_rowid column of MVIEW_EXCEPTIONS gives the table rowids with the discrepancy (i.e., all rows corresponding to the violated relationship). In our example, it would return all rows corresponding to the towns of London and Liverpool. We can now look at the actual data values by looking up the customer table, as follows. We can see that customer_ids AB130000 and AB130001 have mismatched values for town, postal_code, and county with respect to the remaining data.

SELECT customer_id, town, county, postal_code FROM customer
WHERE rowid IN (select bad_rowid FROM system.mview_exceptions);

CUSTOMER_I TOWN       COUNTY     POSTAL_COD
---------- ---------- ---------- ----------
AB123477   London     London     W1 1QC
AB123478   London     London     W1 1QC
AB123480   London     London     W1 1QC
...
AB130000   London     London     W1 2QC    <- bad data
AB123496   Liverpool  Mersey     LV1 1QT
AB123497   Liverpool  Mersey     LV1 1QT
AB123498   Liverpool  Mersey     LV1 1QT
...
AB130001   Liverpool  Hants      LV1 1QT   <- bad data

In the next section, we will look at how query rewrite can be used to transparently rewrite queries with materialized views. We will also see how dimension objects are used.


Query rewrite

4.6 Query rewrite

The greatest benefit of using Summary Management in Oracle is automatic query rewrite. Traditionally, when summary tables were used, the queries on the base tables had to be modified to use the summary table instead. However, with query rewrite, the Oracle optimizer automatically determines if the result of a query can be obtained from a materialized view and, if so, transparently rewrites the query in terms of the materialized view. If the optimizer determines that the summary is insufficient to answer the query, it uses the detail data. This means that no application changes are needed in order to create and use summaries. Further, end users do not have to be aware of the existence of summaries and hence summaries can be created and modified without impacting users.

Oracle supports several types of query rewrite transformations, allowing a single summary to be used to answer several queries. We will illustrate several of these in the following sections, including:

  • Text match

  • Aggregate rollup

  • Join-back

  • Filtered data

  • Rollup using a dimension object

To enable queries to be rewritten, the materialized view must be created with the ENABLE QUERY REWRITE clause. Also, the QUERY_REWRITE_ENABLED initialization parameter must be set to TRUE and the cost-based optimizer must be used.

-- enable query rewrite
ALTER SESSION SET QUERY_REWRITE_ENABLED=TRUE;

-- enable cost-based optimizer
ALTER SESSION SET OPTIMIZER_MODE=" all_rows";

Our first few examples will use the following materialized view, which computes the sum of sales for products by month.

CREATE MATERIALIZED VIEW monthly_sales_mv
 ENABLE QUERY REWRITE
 AS
 SELECT t.month, t.year, p.product_id,
        SUM (ps.purchase_price) as sum_of_sales,
        COUNT (ps.purchase_price) as total_sales
FROM time t, product p, purchases ps
WHERE t.time_key = ps.time_key AND
      ps.product_id = p.product_id
GROUP BY t.month, t.year, p.product_id;

4.6.1 How can we tell if a query has been rewritten?

To determine if the query has been rewritten, use EXPLAIN PLAN (described in Chapter 3) to look at the query execution plan. The output of EXPLAIN PLAN shows the name of the materialized view used to rewrite the query. We will use this in the examples in this section. Alternatively, in SQLPLUS, you can use the SET AUTOTRACE option to trace the optimizer plan. If you find that query rewrite is not occurring as expected, you could use the DBMS_MVIEW.EXPLAIN_REWRITE utility discussed later in this chapter to diagnose the problem.

Hint:

Name your materialized views with a _mv suffix. This will help quickly identify them in the execution plan output.

4.6.2 SQL text match

The simplest type of query rewrite is when the SQL text of the materialized view's defining query exactly matches that of the incoming query. The text match is not case sensitive and ignores any whitespace differences.

The execution plan in the following code segment shows that the optimizer chose to access the materialized view MONTHLY_SALES_MV via a full table scan.

-- exact text match
SELECT t.year, t.month, p.product_id,
       sum (ps.purchase_price) as sum_of_sales,
       count (ps.purchase_price) as total_sales
FROM time t, product p, purchases ps
WHERE t.time_key   = f.time_key AND
f.product_id = p.product_id
GROUP BY t.year, t.month, p.product_id;

 MONTH       YEAR     PRODUCT_ SUM_OF_SALES TOTAL_SALES
---------- ---------- -------- ------------ -----------
         1       2002 SP1000       54966.53         332
         1       2002 SP1001       48861.17         312
         1       2002 SP1010       45432.43         268
...
-----------------------------------------------------------------
|Id| Operation          |  Name           | Rows  | Bytes |Cost |
-----------------------------------------------------------------
| 0| SELECT STATEMENT   |                 |   327 | 18966 |   2 |
| 1|  TABLE ACCESS FULL | MONTHLY_SALES_MV|   327 | 18966 |   2 |
-----------------------------------------------------------------

Oracle will also try a text match starting from the FROM keyword of the query. This allows for differences in column ordering in the SELECT list and computation of expressions. In the following example, SUM(purchase_price) and COUNT(purchase_price) have been used to compute the average, and the order of t.month and t.year in the SELECT list is reversed. You can see that the materialized view has been used to rewrite this query.

SELECT t.month, t.year, p.product_id,
       AVG(ps.purchase_price) avg_of_sales
FROM time t, product p, purchases ps
WHERE t.time_key = ps.time_key AND
      ps.product_id = p.product_id
GROUP BY t.year, t.month, p.product_id;

MONTH       YEAR      PRODUCT_ID AVG_OF_SALES
---------- ---------- ---------- ------------
         1       2002 SP1000       165.561837
         1       2002 SP1001       156.606314
         1       2002 SP1010       169.523993
...

-----------------------------------------------------------------
|Id| Operation          |  Name           | Rows  | Bytes |Cost |
-----------------------------------------------------------------
| 0| SELECT STATEMENT   |                 |   327 | 18966 |   2 |
| 1|  TABLE ACCESS FULL | MONTHLY_SALES_MV|   327 | 18966 |   2 |
-----------------------------------------------------------------

If the text of the query and materialized view does not match, Oracle will then compare the join conditions, GROUP BY clauses, and aggregates in the query and materialized view to determine if the query can be rewritten using the materialized view. We will illustrate these rules in the following sections.

4.6.3 Aggregate rollup

An aggregate rollup (often simply called rollup) occurs when the aggregates in the materialized view can be further aggregated to determine the aggregates requested by the query. A simple example of this is when the query contains only some of the grouping columns from the materialized view. For instance, if the materialized view groups by year and month, and the query asks for year, then the aggregates in the materialized view for each month for a given year are aggregated to produce one row for the year.

-- rollup over month column
SELECT t.year, p.product_id,
       SUM (ps.purchase_price) as sum_of_sales,
       COUNT (ps.purchase_price) as total_sales
FROM time t, product p, purchases ps
WHERE t.time_key = ps.time_key AND
      ps.product_id = p.product_id
GROUP BY t.year, p.product_id;

YEAR       PRODUCT_ID SUM_OF_SALES TOTAL_SALES
---------- ---------- ------------ -----------
      2002 SP1000        142173.51         659
      2002 SP1001         97910.04         475
      2002 SP1010        122422.72         514
...

------------------------------------------------------------------
|Id| Operation          |  Name           | Rows  | Bytes | Cost |
------------------------------------------------------------------
| 0| SELECT STATEMENT   |                 |   327 | 14715 |    5 |
| 1|  SORT GROUP BY     |                 |   327 | 14715 |    5 |
| 2|   TABLE ACCESS FULL| MONTHLY_SALES_MV|   327 | 14715 |    2 |
------------------------------------------------------------------

When a rollup occurs, the rewritten query will contain a GROUP BY clause, as highlighted in the previous execution plan.

A more interesting case of rollup is when your data has a hierarchy described by a dimension object. In this case, query rewrite can rollup data from a lower level to a higher level in the hierarchy. We will explain this later in this chapter.

4.6.4 Join-back

For query rewrite to occur, all the columns in the query must either appear in the materialized view or must be derivable from some column in the materialized view. In the latter case, the materialized view must be joined to the base table to obtain the required column. This is called a join-back. In the simple case, for a join-back to occur, the materialized view must contain either the primary key or the rowid of the detail table. For instance, product_id is the primary key of the product table. So the following query, asking for product_name, can be answered with the MONTHLY_SALES_MV materialized view, using a join-back. The optimizer's plan shows that the query has been rewritten to use the materialized view MONTHLY_SALES_MV with a join to the product table.

-- join-back to product table using primary key constraint
SELECT t.year, t.month, p.product_name,
       SUM (ps.purchase_price) as sum_of_sales
FROM time t, product p, purchases ps
 WHERE t.time_key = ps.time_key AND
       ps.product_id = p.product_id
 GROUP BY t.year, t.month, p.product_name;

YEAR       MONTH      PRODUCT_NAME  SUM_OF_SALES
---------- ---------- ------------- ------------
      2002          1 CD LX1        40208.33
      2002          1 CD LX2        54031.52
      2002          1 CD LX3         9145.54
...

------------------------------------------------------------------
|Id| Operation           |  Name           |Rows  | Bytes | Cost |
------------------------------------------------------------------
| 0| SELECT STATEMENT    |                 |  157 |  9734 |    9 |
| 1|  SORT GROUP BY      |                 |  157 |  9734 |    9 |
| 2|   HASH JOIN         |                 |  327 | 20274 |    5 |
| 3|    TABLE ACCESS FULL| PRODUCT         |  162 |  2754 |    2 |
| 4|    TABLE ACCESS FULL| MONTHLY_SALES_MV|  327 | 14715 |    2 |
------------------------------------------------------------------

4.6.5 Filtered data

The materialized view MONTHLY_SALES_MV defined previously contained data for all products for each month and year. Sometimes you may only want to summarize data for a certain product, region, or year. In this case, the materialized view will only contain a subset of data indicated by a selection condition in the WHERE clause of its query. In Oracle 9i, sophisticated query rewrites are possible with such a materialized view. Oracle will determine if the query can be answered by the materialized view by analyzing and comparing the WHERE clauses of the materialized view and the query.

The following materialized view contains sum of sales and number of sales for the electronics category for the years from 1997 through 2002.

CREATE MATERIALIZED VIEW elec_sales_mv
ENABLE QUERY REWRITE
AS
SELECT t.month, t.year, p.product_id,
       SUM(ps.purchase_price) as sum_of_sales,
       COUNT (ps.purchase_price) as total_sales
FROM time t, product p, purchases ps
WHERE t.time_key = ps.purchase_date AND
      ps.product_id = p.product_id AND
      p.category = 'ELEC' AND t.year BETWEEN 1997 and 2002
GROUP BY t.month, t.year, p.product_id;

This materialized view can be used to answer the following query, which requests the sum of sales and number of sales for the electronics category for 1999. Notice that following the execution plan is a predicate information section. This is part of the EXPLAIN PLAN output and shows the predicates applied during each step of the execution plan. We show it here to illustrate that upon rewrite, the MV data is filtered to only select the row for 1999.

SELECT t.month, p.product_id,
   SUM(ps.purchase_price) as sum_of_sales,
   COUNT (ps.purchase_price) as total_sales
 FROM time t, product p, purchases ps
 WHERE t.time_key = ps.purchase_date AND
   ps.product_id = p.product_id AND
   p.category = 'ELEC' AND t.year = 1999
 GROUP BY t.month, t.year, p.product_id;

MONTH      PRODUCT_ SUM_OF_SALES TOTAL_SALES
---------- -------- ------------ -----------
         1 SP1000       54966.53         332
         1 SP1001       48861.17         312
         1 SP1010       45432.43         268
...

----------------------------------------------------------------
|Id| Operation          |  Name         | Rows  | Bytes | Cost |
----------------------------------------------------------------
| 0| SELECT STATEMENT   |               |     2 |   116 |    2 |
|*1|  TABLE ACCESS FULL | ELEC_SALES_MV |     2 |   116 |    2 |
----------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter("ELEC_SALES_MV"."YEAR"=1999)

On the other hand, the answer to a query that requests the sum of sales for the year 1999 for all categories must be obtained from the detail tables, because the materialized view does not have sufficient data to compute sales for product categories other than electronics.

A query can have additional conditions not mentioned in the materialized view. For instance in the following query, we are looking for monthly sales of digital camera products in the electronics category for 2002. The materialized view has all products within this category, and we can determine the product_name from product_id using a join-back. Hence, the query will be rewritten to use the materialized view with a join to the product table.

SELECT t.month, SUM(ps.purchase_price) as sum_of_sales,
   COUNT (ps.purchase_price) as total_sales
 FROM time t, product p, purchases ps
 WHERE t.time_key = ps.purchase_date AND
   ps.product_id = p.product_id AND p.category = 'ELEC' AND
   t.year = 2002 AND product_name  = 'Digital Camera'
GROUP BY t.month;

MONTH      SUM_OF_SALES TOTAL_SALES
---------- ------------ -----------
         1     54966.53         332
         2     32861.22         204
         4     24575.66          82
         6     29770.1           41

---------------------------------------------------------------
|Id| Operation           |  Name           | Rows |Bytes| Cost|
---------------------------------------------------------------
| 0| SELECT STATEMENT    |                 |    1 |  75 |   4 |
| 1|  TABLE ACCESS BY    | PRODUCT         |    1 |  17 |   1 |
|  |         INDEX ROWID |                 |      |     |     |
| 2|   NESTED LOOPS      |                 |    1 |  75 |   4 |
| 3|    TABLE ACCESS FULL| ELEC_SALES_MV   |    2 | 116 |   2 |
| 4|    INDEX RANGE SCAN | PRODUCT_PK_INDEX|  162 |     |     |
---------------------------------------------------------------

This example illustrates how two rewrite mechanisms can be applied together-join-back and filtered data.

4.6.6 Computing other aggregates in the query

Aggregates in the query can be computed from different aggregates in the materialized view. We already saw a simple example of this in the SQL text match section. However, the power of query rewrite comes from the fact that many different transformations can be combined together. For instance, in the following query, we want to know the average purchase price of each item by year. The materialized view has the sum and count of the purchase_price at the month, year level. However, the average can be computed by first doing a rollup of months to years and then dividing the sum by the count of the purchase_price. The query is therefore rewritten to use the MONTHLY_SALES_MV materialized view

-- aggregate computability

SELECT t.year, p.product_id, AVG(ps.purchase_price) as ave_sales
FROM time t, product p, purchases ps
WHERE t.time_key = ps.time_key AND
ps.product_id = p.product_id
GROUP BY  t.year, p.product_id;

YEAR       PRODUCT_  AVE_SALES
---------- -------- ----------
      2002 SP1000    215.74129
      2002 SP1001     206.1264
      2002 SP1010   238.176498
...

------------------------------------------------------------------
|Id| Operation          |  Name            | Rows | Bytes | Cost |
------------------------------------------------------------------
| 0| SELECT STATEMENT   |                  |  327 | 14715 |    5 |
| 1|  SORT GROUP BY     |                  |  327 | 14715 |    5 |
| 2|   TABLE ACCESS FULL| MONTHLY_SALES_MV |  327 | 14715 |    2 |
------------------------------------------------------------------

To compute aggregates in the query with a rollup, the materialized view may need to have auxiliary aggregates. For instance, to roll up AVG, the materialized view must have SUM and COUNT or AVG and COUNT.

All the query rewrites we have seen so far have not required any additional information from the user. However, to get the most out of query rewrite, you must inform query rewrite about relationships between data columns using constraints and dimensions.

4.6.7 Rewrite using dimensions

One of the powerful features of query rewrite is the ability for a single materialized view to be used to satisfy a wide range of queries. The dimension object discussed in section 4.5, is extremely useful in this respect. By allowing you to declare relationships within columns of dimension tables, it provides query rewrite with information to roll up from a lower to a higher level of a hierarchy. For example, suppose your users may want to know the sum of sales by day, month, or year. You could create three materialized views to answer these queries, or you could create one at the level of day and then define a dimension object to show the relationship between day, month, and year. Now, when the query asks for data at the monthly level, the materialized view at the daily level will be used to roll up the data to the monthly level.

Recall that the relationships declared by dimensions are not validated automatically by Oracle. Hence, you must set the QUERY_REWRITE_INTEGRITY parameter to TRUSTED or STALE_TOLERATED, indicating that the relationships stated by the dimension object can be trusted.

Consider the following definition for the time dimension. The HIERARCHY clause tells query rewrite that the time_key rolls up into week_number, which, in turn, rolls up into quarter. This means that if we knew the time_key value for some row in the TIME table, we know which week it belonged to. Similarly, if we knew the month (say Jan 2002), we know the year it belonged to: 2002.

CREATE DIMENSION time
  LEVEL time_key        is time.time_key
  LEVEL month           is time.month
  LEVEL quarter         is time.quarter
  LEVEL year            is time.year
  LEVEL week_number     is time.week_number
  HIERARCHY fiscal_rollup (
            time_key                CHILD OF
            week_number             CHILD OF
            quarter)
  HIERARCHY calendar_rollup(
            time_key                CHILD OF
            month                   CHILD OF
            year);

Suppose our materialized view MONTHLY_SALES_MV was defined to report the total sales by product and month, as follows:

CREATE MATERIALIZED VIEW monthly_sales_mv
 ENABLE QUERY REWRITE
 AS
 SELECT t.month, p.product_id,
        SUM(ps.purchase_price) as sum_of_sales,
        COUNT (ps.purchase_price) as total_sales
 FROM time t, product p, purchases ps
 WHERE t.time_key = ps.time_key AND
       ps.product_id = p.product_id
 GROUP BY t.month, p.product_id;

In the following query, we want to know the total sales by product by year. Since we have a materialized view with the total sales by product by month, and months can be rolled up into years, as specified in the calendar_rollup hierarchy in the time dimension, the optimizer will rewrite the query to use the materialized view, MONTHLY_SALES_MV. Note that in order to determine the year value for the month, Oracle will do a join-back from the materialized view to the time table.

-- rollup to higher LEVEL in the HIERARCHY
SELECT t.year, p.product_id,
       COUNT (ps.purchase_price) as total_sales
FROM time t, product p, purchases ps
WHERE t.time_key = ps.time_key AND
      ps.product_id = p.product_id
GROUP BY t.year, p.product_id;

------------------------------------------------------------------
|Id| Operation             | Name           |Rows  |Bytes | Cost |
------------------------------------------------------------------
| 0| SELECT STATEMENT      |                |    1 |   39 |    9 |
| 1|  SORT GROUP BY        |                |    1 |   39 |    9 |
| 2|   HASH JOIN           |                |  176 | 6864 |    7 |
| 3|    VIEW               |                |    5 |   35 |    4 |
| 4|     SORT UNIQUE       |                |    5 |   35 |    4 |
| 5|      TABLE ACCESS FULL|TIME            |   15 |  105 |    2 |
| 6|    TABLE ACCESS FULL  |MONTHLY_SALES_MV|  246 | 7872 |    2 |
------------------------------------------------------------------

Using the ATTRIBUTE clause in a dimension

When a data warehouse is being defined, the dimension object is often overlooked, because its value to query rewrite is not fully appreciated. However, a dimension object gives you tremendous rewrite power at no extra cost. We have already seen how query rewrite can take advantage of the HIERARCHY clause to rewrite several queries with one materialized view. Query rewrite can also make use of the ATTRIBUTE clause of dimension. In the following example, we want to know the sum of sales by customers based on their gender and occupation.

SELECT c.sex, c.occupation, SUM(ps.purchase_price) as sum_of_sales
FROM purchases ps,  customer c
WHERE c.customer_id = ps.customer_id
GROUP BY c.sex, c.occupation;

We could have put the columns sex and occupation into a materialized view. But we know that given the customer_id, we can find information such as the customer's name, sex, and occupation. Such relationships within a table that are not hierarchical in nature are defined by the ATTRIBUTE clause in a dimension. The ATTRIBUTE clause tells us that each customer_id has a one-to-one relationship to the customer's name, sex, and occupation.

The following dimension defines the relationships within the customer table.

CREATE DIMENSION customer_dim
  LEVEL customer IS customer.customer_id
  LEVEL town     IS customer.town
  LEVEL region   IS customer.county
HIERARCHY customer_zone
  ( customer CHILD OF
    town     CHILD OF
    region )
ATTRIBUTE customer DETERMINES (sex, occupation);

Now that we have this dimension object, we only need to include the customer_id in the materialized view.

CREATE MATERIALIZED VIEW cust_sales_mv
ENABLE QUERY REWRITE
AS
SELECT c.customer_id, SUM(ps.purchase_price) as sum_of_sales,
       COUNT (ps.purchase_price) as total_sales
FROM customer c, purchases ps
WHERE c.customer_id = ps.customer_id
GROUP BY c.customer_id;

The execution plan of the query shows that the query was rewritten to use the materialized view. Note that a join-back was done to the customer table to retrieve the values of the occupation and sex columns.

-----------------------------------------------------------------
|Id | Operation            |  Name       | Rows  | Bytes | Cost |
-----------------------------------------------------------------
| 0 | SELECT STATEMENT     |             |     8 |   320 |    7 |
| 1 |  SORT GROUP BY       |             |     8 |   320 |    7 |
| 2 |   HASH JOIN          |             |    82 |  3280 |    5 |
| 3 |    TABLE ACCESS FULL | CUSTOMER    |    40 |   800 |    2 |
| 4 |    TABLE ACCESS FULL | CUST_SALES  |    82 |  1640 |    2 |
----------------------------------------------------------------|

When designing your data warehouse, you should try to identify relationships between your dimension tables and define dimension objects, wherever possible. This will lead to significant space savings and increase query rewrite opportunities, thereby improving your query performance.

4.6.8 Rewrite and constraints

In OLTP systems, constraints are used to enforce data integrity. In a data warehouse, constraints may be used to define the relationships between the fact and dimension tables. Typically, a primary-key constraint is defined on the unique key column on each dimension table. A foreign-key constraint and a NOT NULL constraint are defined on each corresponding key in the fact table.

As discussed in Chapter 2, the EASYDW schema has primary-key constraints on each of the dimension tables-customer, product, and time. Also, there are foreign key and NOT NULL constraints on columns in the purchases table that join to these dimension tables.

The relationship defined by these constraints indicates to query rewrite that a join between the purchases table and, for example, the TIME table, will produce a row for every row in the purchases table. Such a join is known as a lossless join, because no rows in the purchases table will be lost (or duplicated) by the join process. You can use a materialized view to rewrite a query that has fewer joins if the extra joins in the materialized view are lossless. For instance, we can rewrite the following query with the MONTHLY_SALES_MV materialized view even though the time table is omitted completely! This is because the extra join between the PURCHASES and TIME tables in the materialized view is a lossless join.

SELECT  p.product_id, SUM(ps.purchase_price) as sum_of_sales
FROM  product p, purchases ps
WHERE ps.product_id = p.product_id
GROUP BY p.product_id;

PRODUCT_ SUM_OF_SALES
-------- ------------
SP1000      142324.24
SP1001      104369.24
SP1010      122422.72
...
------------------------------------------------------------------
|Id| Operation          |  Name           | Rows  | Bytes | Cost |
------------------------------------------------------------------
| 0| SELECT STATEMENT   |                 |   327 | 10464 |    4 |
| 1|  SORT GROUP BY     |                 |   327 | 10464 |    4 |
| 2|   TABLE ACCESS FULL| MONTHLY_SALES_MV|   327 | 10464 |    2 |
------------------------------------------------------------------

As described in Chapter 2, when a constraint is enabled, you can choose to have Oracle validate the integrity of the data. Typically, in a data ware-house, you may have already validated the data on the operational system or during the ETL process. In this case, you could use the NOVALIDATE clause to tell Oracle that the data has already been validated.

ALTER TABLE purchases ENABLE NOVALIDATE CONSTRAINT fk_customer_id;

Declaring the constraint informs query rewrite about the relationship between the fact and dimension tables. An additional RELY clause should be used to tell Oracle that it can rely upon the constraints being correct and can use it in query rewrite even when the constraint has not been validated. It allows the database administrator (DBA) to say: Trust me. I've already checked the data validity. Query rewrite can rely on the relationship being correct.

ALTER TABLE purchases MODIFY CONSTRAINT fk_customer_id RELY;
Hint:

Use constraints to define the relationship between your fact and dimension tables. Use the dimension object to declare the relationships within your dimension tables, such as a time or a region hierarchy.

4.6.9 Query rewrite integrity modes

We have mentioned the QUERY_REWRITE_INTEGRITY parameter several times in our examples. This parameter indicates to Oracle the extent to which the rewritten queries must reflect the data in the detail tables. This parameter can take three values:

  • ENFORCED

  • STALE_TOLERATED

  • TRUSTED

If the QUERY_REWRITE_INTEGRITY mode is set to ENFORCED (which is the default), Oracle will guarantee that the rewritten query will return the same results as the query without query rewrite. If it is set to STALE_TOLERATED, Oracle will use stale materialized views that do not contain all the detail data. If it is set to TRUSTED, Oracle will use data and relationships that have been "blessed" by the DBA, namely dimension objects, RELY constraints, and materialized views created from PREBUILT tables. Oracle does not validate that the relationships declared by the dimension are valid or that a prebuilt table is the same as the materialized view's query.

You must decide what query rewrite integrity level is appropriate for your application. Most of the time, you would like to get the result the fastest way possible, rewriting your queries to use materialized views. However, if your materialized views have become stale and no longer represent the summarization of all your detail data, you may prefer to get the results from the detail tables until you can perform your next refresh.

Hint:

When first testing a materialized view to see if query rewrite will occur, set QUERY_REWRITE_INTEGRITY to STALE_TOLERATED, because if the query does not rewrite in this mode, it will not rewrite in any other mode. Once you know it works, you can try setting the parameter mode to your desired level.

If the results obtained from a materialized view are close enough for your application, you may want to use the materialized view even if it is stale. To determine the month-over-month growth rate of online sales, you do not need every single sales transaction in the materialized view. As long as the data is reasonably recent, you could still get an answer that was close enough. Or, if the application knew that the missing data was beyond the scope of the query, it may still want to use the materialized view. For instance, if the missing data is for the last month but your query does not need this data, you can use the materialized view. Or, it may be appropriate to use the materialized view when the fact table is stale but not when a dimension is updated.

To tell the optimizer that you'd like to use the materialized view even if it is stale, set the QUERY_REWRITE_INTEGRITY parameter to STALE_TOLERATED. You can determine if a materialized view is FRESH or STALE using the STALENESS column of the catalog view USER_MVIEWS.

The following example shows the difference between the two integrity modes STALE_TOLERATED and ENFORCED with regard to STALE data. We insert four new rows into the purchases fact table.

INSERT INTO purchases VALUES ( 'SP1061','1-FEB-2002',
     'AB123456','1-FEB-2002', 0024,28.01, 4.50, 'Y');

INSERT INTO purchases VALUES ('SP1062','1-FEB-2002',
     'AB123457','1-FEB-2002', 1024,28.01, 4.50, 'Y');

INSERT INTO purchases VALUES ('SP1063','2-FEB-2002',
     'AB123457','2-FEB-2002', 0024,28.01, 4.50, 'N');

INSERT INTO purchases VALUES ('SP1064','2-FEB-2002',
     'AB123457','2-FEB-2002', 1024,28.01, 4.50, 'N');
COMMIT;

The MONTHLY_SALES_MV materialized view is now stale. All the data in the detail table is not reflected in the materialized view.

SELECT staleness FROM user_mviews
WHERE mview_name = 'MONTHLY_SALES_MV';

STALENESS
-------------------
STALE

If you set QUERY_REWRITE_INTEGRITY to ENFORCED (or TRUSTED), Oracle will use the detail tables rather than the materialized view and the sales numbers are updated to include the new rows.

ALTER SESSION SET QUERY_REWRITE_INTEGRITY=ENFORCED;

SELECT t.month, p.product_id,
   SUM(ps.purchase_price) as sum_of_sales,
   COUNT(ps.purchase_price) as total_sales
FROM time t, product p, purchases ps
WHERE t.time_key = ps.time_key AND
  ps.product_id = p.product_id
GROUP BY t.month, p.product_id;

     MONTH PRODUCT_  SUM_OF_SALES TOTAL_SALES
---------- --------  ------------ -----------
         1  SP1000       19683.57         209
         1  SP1001       17377.68         189
         1  SP1010       12500.83         145
         1  SP1011       25094.03         249
         ...
         2  SP1061      362092.71         248    <- new value
         2  SP1062        54775.9         292
         2  SP1063       45862.38         286
         2  SP1064       53637.11         248
         ...

------------------------------------------------------------------
|Id| Operation              | Name           | Rows  |Bytes| Cost|
------------------------------------------------------------------
|0 | SELECT STATEMENT       |                |   567 |23814|  415|
|1 |  SORT GROUP BY         |                |   567 |23814|  415|
|2 |   NESTED LOOPS         |                | 70585 |2895K|   43|
|3 |    HASH JOIN           |                | 70585 |2412K|   43|
|4 |     TABLE ACCESS FULL  |TIME            |    15 |  225|    2|
|5 |     PARTITION RANGE ALL|                |       |     |     |
|6 |      TABLE ACCESS FULL |PURCHASES       | 94113 |1838K|   40|
|7 |    INDEX RANGE SCAN    |PRODUCT_PK_INDEX|     1 |    7|     |
------------------------------------------------------------------

Note that the TRUSTED mode will behave the same as ENFORCED with regard to STALE data-it will not use a STALE materialized view. However, if you can tolerate stale data and set QUERY_REWRITE_INTEGRITY to STALE_TOLERATED, then the materialized view is now used. However, the new rows that were inserted are not reflected in the total!

ALTER SESSION SET QUERY_REWRITE_INTEGRITY=STALE_TOLERATED;

SELECT t.month, p.product_id,
   SUM(ps.purchase_price) as sum_of_sales,
   COUNT(ps.purchase_price) as total_sales
FROM time t, product p, purchases ps
WHERE t.time_key = ps.time_key AND
    ps.product_id = p.product_id
GROUP BY t.month, p.product_id;

MONTH      PRODUCT_  SUM_OF_SALES TOTAL_SALES
---------- --------  ------------ -----------
         1 SP1000        19683.57         209
         1 SP1001        17377.68         189
         1 SP1010        12500.83         145
         1 SP1011        25094.03         249
         ...
         2 SP1061        362064.7         247   <- old value
         2 SP1062        54747.89         291
         2 SP1063        45834.37         285
         2 SP1064         53609.1         247
         ...
------------------------------------------------------------------
|Id| Operation          |  Name            | Rows  | Bytes | Cost|
------------------------------------------------------------------
| 0| SELECT STATEMENT   |                  |   327 | 18966 |    2|
| 1|  TABLE ACCESS FULL | MONTHLY_SALES_MV |   327 | 18966 |    2|
------------------------------------------------------------------

You are probably thinking that setting the parameter to ENFORCED is the best mode to use, since it guarantees that you will see the same results from using the materialized view or querying the detail tables. The problem is that in ENFORCED mode, all constraints must be validated and dimension objects are not used. Unless you have validated all the defined relationships, such as constraints, query rewrite may never occur. Therefore, in a warehouse you will most likely use the TRUSTED mode. In this mode, materialized views are only used if they are fresh; constraints are used provided they have the RELY clause, and dimensions are also considered, even though they are not validated.

Caution

Query rewrite depends on the integrity of your dimension and constraint definitions. Does each product in the product table roll up to one, and only one, category, as specified in your dimension definition? Does each product in the purchases table have a corresponding product_id in the products table, as specified by your referential integrity constraints? If your data do not reflect the relationships defined by the constraint or dimension, you may get unexpected results. The same holds for materialized views on prebuilt tables. If the prebuilt table does not reflect the materialized view's query accurately, then results can be unexpected. To summarize, TRUSTED is the recommended mode to use in a warehouse, but you must ensure that all data is correct.

4.6.10 Query rewrite and the optimizer

Query rewrite is only available when using the cost-based optimizer. While it is normally faster to obtain the results from a materialized view, there may be cases where it is faster to retrieve the results from the detail table. The query optimizer compares the cost of the execution plan with and without query rewrite and uses the one with the lower cost. To ensure that the optimizer makes the correct choices, you need to collect statistics on the detail tables involved in the query and on the materialized views using the DBMS_STATS package.

However, if your application requires that the query must use the materialized view and must not use the base tables, you can set the QUERY_REWRITE_ENABLED to FORCE. In this mode, if there is a materialized view that satisfies the query, the optimizer will use it. It will not compare the cost of plan with and without rewrite. This parameter value is new in Oracle 9i Release 2.

If several materialized views are eligible to rewrite the query, query rewrite will choose the one that would need to read less data. You can, however, request rewrite to use a specific materialized view by using the /*+ REWRITE(mv) */ hint. You can disable query rewrite for a statement by using the /*+ NOREWRITE */ hint. For instance, suppose we had two eligible materialized views: MONTHLY_SALES_MV, which computes sum of sales by month, and YEARLY_SALES_MV, which computes the sum of sales by year. For the following query, you would expect query rewrite to pick the latter, since it would be smaller. You could, however, force query rewrite to use MONTHLY_SALES_MV with a hint.

SELECT t.year, p.product_id, SUM(ps.purchase_price) sum_of_sales
FROM time t, product p, purchases ps
WHERE t.time_key = ps.time_key AND
      ps.product_id = p.product_id
GROUP BY t.year, p.product_id;

------------------------------------------------------------------
|Id| Operation         |  Name           | Rows  | Bytes | Cost  |
------------------------------------------------------------------
| 0| SELECT STATEMENT  |                 |    82 |  2624 |     2 |
| 1|  TABLE ACCESS FULL| YEARLY_SALES_MV |    82 |  2624 |     2 |
------------------------------------------------------------------

SELECT /*+ REWRITE(monthly_sales_mv) */ t.year, p.product_id,
       SUM(ps.purchase_price) as sum_of_sales
FROM time t, product p, purchases ps
WHERE t.time_key = ps.time_key AND
      ps.product_id = p.product_id
GROUP BY t.year, p.product_id;

------------------------------------------------------------------
|Id| Operation             | Name           | Rows | Bytes | Cost|
------------------------------------------------------------------
| 0| SELECT STATEMENT      |                |  115 |  2415 |  10 |
| 1|  SORT GROUP BY        |                |  115 |  2415 |  10 |
| 2|   HASH JOIN           |                |  589 | 12369 |   7 |
| 3|    VIEW               |                |    5 |    35 |   4 |
| 4|     SORT UNIQUE       |                |    5 |    35 |   4 |
| 5|      TABLE ACCESS FULL|TIME            |   15 |   105 |   2 |
| 6|    TABLE ACCESS FULL  |MONTHLY_SALES_MV|  825 | 11550 |   2 |
------------------------------------------------------------------

4.6.11 Query rewrite and partition change tracking

Previously, we discussed the partition change tracking feature in Oracle 9i that allows materialized views to be fast refreshed after partition maintenance operations. Partition change tracking also increases the query rewrite capabilities of the materialized view. Normally, when a detail table is updated, the materialized view becomes stale and cannot be used by query rewrite in ENFORCED or TRUSTED integrity levels. However, if the detail table is partitioned and the materialized view supports partition change tracking on that table (by including the partition key or partition marker for that table), Oracle can determine if a portion of the materialized view is fresh. If a query can be answered only using the fresh portion of the materialized view, query rewrite will use the materialized view. For example, in Figure 4.4 (see section 4.3.3) the fresh portion of the materialized view corresponds to the Jan 2002 and Mar 2002 partitions. The materialized view cannot be used for Feb 2002 (updated partition) or Apr 2002 (new partition).

Consider the following materialized view containing sales data for products. The purchases table is partitioned by time_key, which is included in the materialized view.

CREATE MATERIALIZED VIEW product_category_sales_mv
ENABLE QUERY REWRITE
AS
SELECT ps.time_key, p.category,
       SUM(ps.purchase_price) as sum_of_sales
FROM product p, purchases ps
WHERE ps.product_id = p.product_id
GROUP BY ps.time_key, p.category;

If we query the view user_mviews, we will see that the materialized view is FRESH.

SELECT staleness FROM user_mviews
WHERE mview_name = 'PRODUCT_CATEGORY_SALES_MV';

STALENESS
-------------------
FRESH

The materialized view only contains data through Mar 2002. Now, suppose we added a new partition to the purchases table and loaded data for Apr 2002.

ALTER TABLE purchases ADD PARTITION purchases_apr2002
          values less than (TO_DATE('01-05-2002', 'DD-MM-YYYY'));

INSERT INTO purchases VALUES ( 'SP1063','2-APR-2002',
          'AB123457','7-APR-2002', 0024,28.01, 4.50, 'N');

INSERT INTO purchases VALUES ( 'SP1064','2-APR-2002',
          'AB123457','8-APR-2002', 1024,28.01, 4.50, 'N');
...

COMMIT;

If we query the view user_mviews now, we will see that the materialized view is STALE.

SELECT staleness FROM user_mviews
WHERE mview_name = 'PRODUCT_CATEGORY_SALES_MV';

STALENESS
-------------------
STALE

Now, consider the following query, which asks for the sum of sales for January through March 2002. The Oracle optimizer will determine that the query only needs to access partitions for Jan, Feb, and Mar 2002 of the purchases table. Since the materialized view is enabled for partition change tracking for this table, Oracle will determine that the materialized view is fresh with respect to these partitions. Hence, it can rewrite with the materialized view, as shown in the following execution plan.

SELECT ps.time_key, p.category,
       SUM(ps.purchase_price) as sum_of_sales
 FROM  product p, purchases ps
 WHERE ps.product_id = p.product_id and
       ps.time_key  BETWEEN TO_DATE('01-01-2002', 'DD-MM-YYYY')
                        AND TO_DATE('31-03-2002', 'DD-MM-YYYY')
 GROUP BY ps.time_key, p.category;

---------------------------------------------------------------
|Id| Operation         |Name             | Rows| Bytes | Cost |
---------------------------------------------------------------
|0 | SELECT STATEMENT  |                 |   1 |    26 |    2 |
|1 |  TABLE ACCESS FULL|PRODUCT_CATEGORY_|   1 |    26 |    2 |
|  |                   |        SALES_MV |     |       |      |
---------------------------------------------------------------

On the other hand, a query that requests the sum of sales for Apr 2002 is answered using the detail data, as shown in the following example.

SELECT ps.time_key, p.category,
       SUM(ps.purchase_price) as sum_of_sales,
FROM  product p, purchases ps
WHERE ps.product_id = p.product_id and
      ps.time_key  BETWEEN TO_DATE('01-01-2002', 'DD-MM-YYYY')
                       AND TO_DATE('30-04-2002', 'DD-MM-YYYY')
 GROUP BY ps.time_key, p.category;

---------------------------------------------------------------
|Id | Operation             |  Name    | Rows  | Bytes | Cost |
---------------------------------------------------------------
|  0| SELECT STATEMENT      |          |    43 |  1376 |  442 |
|  1|  SORT GROUP BY        |          |    43 |  1376 |  442 |
|  2|   HASH JOIN           |          | 94113 |  2941K|   43 |
|  3|    TABLE ACCESS FULL  | PRODUCT  |   162 |  1944 |    2 |
|  4|    PARTITION RANGE ALL|          |       |       |      |
|  5|     TABLE ACCESS FULL | PURCHASES| 94113 |  1838K|   40 |
---------------------------------------------------------------

4.6.12 Query rewrite and grouping sets

The SQL aggregation operators, CUBE, ROLLUP, and GROUPING SETS, provide a mechanism to compute multiple groupings in a single query. You can create a materialized view using a query with these operators to store multiple levels of aggregation, instead of separate materialized views for each level. Query rewrite can be used to rewrite a query that asks for any of these levels of aggregation.

The following example shows a materialized view, with grouping sets that computes the sum of sales for the three groupings: (category, time_key), (category, time_key, town), and (time_key, county). Note that the materialized view must have a GROUPING_ID or GROUPING function on the group by columns to distinguish rows that correspond to different groupings.

CREATE MATERIALIZED VIEW sales_mv
ENABLE QUERY REWRITE
AS
SELECT p.category, t.time_key, c.county, c.town,
       SUM(f.purchase_price) sales,
       GROUPING_ID(p.category, t.time_key, c.town, c.county) gid
FROM product p, purchases f,  time t, customer c
WHERE p.product_id = f.product_id AND
      t.time_key = f.time_key AND
      c.customer_id = f.customer_id
GROUP BY GROUPING SETS ((p.category, t.time_key),
                        (p.category, t.time_key, c.town),
                        (t.time_key, c.county));

We will now show how this materialized view can be used to rewrite different types of queries.

Rewriting a query with a simple GROUP BY clause

This materialized view can be used to rewrite a query that asks for any grouping that is present in the materialized view or one that can be derived using a rollup. For example, the following query, which asks for total sales by category and time_key, can be rewritten to use the SALES_MV materialized view.

SELECT p.category, t.time_key, SUM(f.purchase_price) sales
FROM product p, purchases f, time t, customer c
WHERE p.product_id = f.product_id AND
      t.time_key = f.time_key AND
      c.customer_id = f.customer_id
GROUP BY p.category, t.time_key;

From the predicate information in the EXPLAIN PLAN output, we see that rewrite was done by selecting rows from SALES_MV with gid = 3, which corresponds to the grouping (p.category, t.time_key).

---------------------------------------------------------------
| Id| Operation          |  Name      | Rows  | Bytes | Cost  |
---------------------------------------------------------------
|  0| SELECT STATEMENT   |            |     1 |    39 |     2 |
|* 1|  TABLE ACCESS FULL | SALES_MV   |     1 |    39 |     2 |
---------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("SALES_MV"."GID"=3)

Rewriting a query with multiple groupings

If the query has multiple groupings, Oracle will try to find a materialized view that satisfies all the groupings. For instance, the following query can be rewritten to use the SALES_MV materialized view defined previously. The output of EXPLAIN PLAN indicates that rewrite used two groupings from the materialized views: gid = 3, corresponding to (p.category, t.time_key), and gid = 10, corresponding to (t.time_key, c.county).

SELECT p.category,  t.time_key, c.county,
       SUM(f.purchase_price) sales
FROM product p, purchases f,  time t, customer c
WHERE p.product_id = f.product_id AND
      t.time_key = f.time_key AND
      c.customer_id = f.customer_id
GROUP BY GROUPING SETS ((p.category, t.time_key),
      (t.time_key, c.county));

-----------------------------------------------------------------
| Id| Operation            |  Name       | Rows  | Bytes | Cost |
-----------------------------------------------------------------
|  0| SELECT STATEMENT     |             |     1 |    46 |    2 |
|* 1|  TABLE ACCESS FULL   | SALES_MV    |     1 |    46 |    2 |
-----------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("SALES_MV"."GID"=3 OR "SALES_MV"."GID"=10)

Rewriting a query with multiple materialized views

If Oracle cannot find a single materialized view to answer the query, it will try to rewrite each grouping separately. (Note that a query with grouping sets can be expressed as a UNION ALL of queries with the individual groupings.) As a result, several materialized views may be used to rewrite the query, and some groupings may remain unrewritten and use the detail data. This rewrite transformation was introduced in Oracle 9i release2.

The next query cannot be rewritten using SALES_MV alone. The grouping (p.category, t.time_key) is present in the SALES_MV materialized view, and the grouping (t.time_key, c.town) can be derived using a rollup of (p.category, t.time_key, c.town). However, the grouping (p.category, c.county, t.year) is not present in this materialized view.

SELECT p.category,  t.time_key, c.county, t.year, c.town,
 SUM(f.purchase_price) sales
FROM product p, purchases f,  time t, customer c
WHERE p.product_id = f.product_id AND
      t.time_key = f.time_key AND
      c.customer_id = f.customer_id
GROUP BY GROUPING SETS ((p.category, t.time_key),
            (t.time_key, c.town),
                (p.category, c.county, t.year));

------------------------------------------------------------------
|Id | Operation               |  Name     | Rows  | Bytes | Cost |
------------------------------------------------------------------
|  0| SELECT STATEMENT        |           |     5 |   290 |  120 |
|  1|  VIEW                   |           |     5 |   290 |  120 |
|  2|   UNION-ALL             |           |       |       |      |
|* 3|    TABLE ACCESS FULL    | SALES_MV  |     1 |    39 |    2 |
|  4|    SORT GROUP BY        |           |     1 |    47 |    4 |
|* 5|     TABLE ACCESS FULL   | SALES_MV  |     1 |    47 |    2 |
|  6|    SORT GROUP BY        |           |     3 |   186 |  114 |
|* 7|     HASH JOIN           |           | 13374 |   809K|   16 |
|  8|      TABLE ACCESS FULL  | PRODUCT   |   162 |  1944 |    2 |
|* 9|      HASH JOIN          |           | 13374 |   653K|   13 |
| 10|       TABLE ACCESS FULL | CUSTOMER  |    82 |   820 |    2 |
|*11|       HASH JOIN         |           | 13374 |   522K|   10 |
| 12|        TABLE ACCESS FULL| TIME      |     7 |    84 |    2 |
| 13|        TABLE ACCESS FULL| PURCHASES | 13379 |   365K|    7 |
------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - filter("SALES_MV"."GID"=3)
   5 - filter("SALES_MV"."GID"=1)
   7 - access("P"."PRODUCT_ID"="F"."PRODUCT_ID")
   9 - access("C"."CUSTOMER_ID"="F"."CUSTOMER_ID")
  11 - access("T"."TIME_KEY"="F"."TIME_KEY")

The EXPLAIN PLAN output shows that rewrite was done using SALES_MV for two groupings (gid = 3 and gid = 1) and using the detail tables for grouping (p.category, c,county, t.year).

Now, suppose we had a simple materialized view, SALES_MV2, that had the grouping (p.category, c.county, t.year).

CREATE MATERIALIZED VIEW sales_mv2
ENABLE QUERY REWRITE
AS
SELECT p.category,  c.county, t.year, SUM(f.purchase_price) sales
FROM product p, purchases f, time t, customer  c
WHERE p.product_id = f.product_id AND
      t.time_key = f.time_key AND
      c.customer_id = f.customer_id
GROUP BY p.category, c.county, t.year;

Oracle can now use the combination of SALES_MV and SALES_MV2 to rewrite the query, as shown in the following execution plan.

------------------------------------------------------------------
| Id  | Operation            |  Name     | Rows  | Bytes | Cost  |
------------------------------------------------------------------
|   0 | SELECT STATEMENT     |           |    84 |  4872 |     8 |
|   1 |  VIEW                |           |    84 |  4872 |     8 |
|   2 |   UNION-ALL          |           |       |       |       |
|*  3 |    TABLE ACCESS FULL | SALES_MV  |     1 |    39 |     2 |
|   4 |    SORT GROUP BY     |           |     1 |    47 |     4 |
|*  5 |     TABLE ACCESS FULL| SALES_MV  |     1 |    47 |     2 |
|   6 |    TABLE ACCESS FULL | SALES_MV2 |    82 |  3034 |     2 |
------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - filter("SALES_MV"."GID"=3)
   5 - filter("SALES_MV"."GID"=1)

4.6.13 Troubleshooting query rewrite problems

In the previous examples, we have used EXPLAIN PLAN to see if a query had been rewritten to use a materialized view. However, sometimes you may find that the query did not rewrite with the materialized view as you had expected. The rules governing query rewrite can be extremely complex, and the reasons for not using a materialized view may not be obvious. In some cases, the reason is extremely trivial, such as the parameter QUERY_REWRITE_ENABLED not being set to TRUE. In other cases, the reason could be more subtle such as a constraint not being present or validated, or some column required by the query not being present in the materialized view. Oracle 9i introduced a utility to assist with diagnosing the reasons for such missed rewrites. This utility is the PL/SQL API DBMS_MVIEW.EXPLAIN_REWRITE.

To use EXPLAIN_REWRITE, you provide the query and, optionally, the materialized view it should use to rewrite. The procedure will tell you if the query will use that materialized view and, if not, the reason for not doing the rewrite. Prior to using the procedure, you must create a table named REWRITE_TABLE in your schema, using the script utlxrw.sql in the rdbms/admin directory. The results of EXPLAIN_REWRITE are placed in this table. There is also a varray interface, which allows you to access the results through a PL/SQL program.

We will now illustrate how to use this utility. In the first example, the user forgot to set the QUERY_REWRITE_ENABLED parameter to TRUE. To diagnose the problem you issue EXPLAIN_REWRITE and select the results from the REWRITE_TABLE.

BEGIN
dbms_mview.explain_rewrite('
SELECT t.month, t.year, p.product_id,
       SUM (ps.purchase_price) as sum_of_sales,
       COUNT (ps.purchase_price) as total_sales
 FROM time t, product p, purchases ps
 WHERE t.time_key = ps.time_key AND
 ps.product_id = p.product_id
 GROUP BY t.month, t.year, p.product_id', 'MONTHLY_SALES_MV');
END;
/

SELECT mv_name, message FROM rewrite_table;

MV_NAME                     MESSAGE
-------------------------   --------------------------------------
MONTHLY_SALES_MV   QSM-01001: query rewrite not enabled

EXPLAIN_REWRITE can be used to check why a specific materialized view was not used to rewrite the query. The query in the following example was used earlier to illustrate the use of constraints with query rewrite. The materialized view has an extra join between purchases and time tables. In this example, however, the constraint fk_time is not marked RELY, and so the join between purchases and time cannot be determined to be a lossless join. Hence, query rewrite fails. This can be very difficult to diagnose without this utility.

BEGIN
dbms_mview.explain_rewrite('
SELECT p.product_id,
       SUM (ps.purchase_price) as sum_of_sales,
       COUNT(ps.purchase_price) as total_sales
 FROM product p, purchases ps
 WHERE p.product_id = ps.product_id
 GROUP BY p.product_id', 'MONTHLY_SALES_MV');
END;
/

SELECT mv_name, message FROM rewrite_table;

MV_NAME                     MESSAGE
----------------------      ---------------------------------------
MONTHLY_SALES_MV            QSM-01110: a lossy join in MV,
                            MONTHLY_SALES_MV, between tables,
                            PURCHASES and TIME, not found in query

MONTHLY_SALES_MV            QSM-01053: NORELY referential integrity
                            constraint on table, PURCHASES, in
                            TRUSTED/STALE TOLERATED integrity mode

Now, let us fix this problem and run EXPLAIN_REWRITE again. We will see that the query now rewrites.

ALTER TABLE purchases MODIFY constraint fk_time rely;

BEGIN
dbms_mview.explain_rewrite('
SELECT p.product_id,
       SUM (ps.purchase_price) as sum_of_sales,
       COUNT(ps.purchase_price) as total_sales
 FROM  product p, purchases ps
 WHERE p.product_id = ps.product_id
 GROUP BY p.product_id', 'MONTHLY_SALES_MV');
END;
/

SELECT mv_name, message FROM rewrite_table;

MV_NAME                        MESSAGE
--------------------------     -----------------------------------
MONTHLY_SALES_MV               QSM-01033: query rewritten with
                               materialized view,
                               MONTHLY_SALES_MV

Sometimes query rewrite may be possible with the requested materialized view; however, there may be a better materialized view that can be used. Suppose we create a materialized view, PRODUCT_SALES_EXACT_MATCH, for the query in the previous example, matching its text exactly. Query rewrite now uses this materialized view instead, since it is more optimal. EXPLAIN_REWRITE will tell you that this is the case.

begin
dbms_mview.explain_rewrite('
SELECT p.product_id,
       SUM(ps.purchase_price) as sum_of_sales,
       COUNT(ps.purchase_price) as total_sales
 FROM  product p, purchases ps
 WHERE p.product_id = ps.product_id
 GROUP BY p.product_id', 'MONTHLY_SALES_MV');
end;
/

SELECT mv_name, message FROM rewrite_table;

MV_NAME                        MESSAGE
--------------------------     -----------------------------------
MONTHLY_SALES_MV               QSM-01009: materialized view,
                               PRODUCT_SALES_EXACT_MATCH,
                               matched query text


Summary Advisor

4.7 Summary Advisor

Now that we have discussed all the benefits that materialized views provide, we come to the question: How do you know what materialized views to create? Decision-support applications typically involve a well-defined set of queries. The simplest solution to improve performance of these queries would be to create one materialized view for each query. In practice, the disk space requirements and refresh overhead will soon get prohibitive. With query rewrite you can use the same materialized view to rewrite a large class of queries. However, determining the optimal set of materialized views to create for a bunch of queries can be tricky. The Summary Advisor tool is designed to help with the problem of choosing the materialized views for an application.

The Summary Advisor can be used to determine what materialized views to create for a given set of queries. The Oracle 9i Summary Advisor is tightly integrated with the query rewrite engine and is aware of the various rules for query rewrite. The Summary Advisor will optimize the materialized views to fit into a storage limit provided by the user.

In order to run the Oracle 9i Summary Advisor you must have Java enabled in the database, and statistics must have been collected on all the relevant tables and their columns. If you have constraints and dimensions defined in your schema to express your business rules, the Summary Advisor will make better recommendations, because it can take advantage of the more sophisticated rewrite rules discussed earlier.

There are two ways you can run the summary advisor: using the Summary Advisor Wizard in Oracle Enterprise Manager or using the DBMS_OLAP package from SQL*Plus.

4.7.1 Summary Advisor Wizard

Oracle Enterprise Manager provides a Summary Advisor Wizard, which guides you through the various steps required for analysis. You can access the Summary Advisor Wizard by navigating to the Warehouse node, followed by the Summary Management node, and then choosing Summary Advisor in the Object menu. Once the analysis is finished you can accept some or all of the recommendations and implement them. Let us look at a typical run of the Summary Advisor Wizard.

The first step is to tell the wizard what SQL statements your application uses. This is known as the workload. The workload can be extracted from various sources, such as Oracle's SQL cache, Oracle trace, or a user-defined table. The Summary Advisor will consider query rewrite rules and statement statistics, such as priority, execution time, and frequency, in coming up with the set of materialized views.

Figure 4.9 shows the Summary Advisor Wizard page for loading the workload.

Click To expand
Figure 4.9: Summary Advisor Wizard—loading the workload.

Since the workload may have several statements, the Summary Advisor provides a mechanism to select only relevant statements using a filter. The filter can be on various criteria, such as the executing user, tables referenced in the query, response time, and frequency. The filter can be applied either while collecting the workload or during the recommendation process. In Figure 4.10 we are setting filters only to consider SQL executed by the EASYDW use and on tables CUSTOMERS and PURCHASES. Next you have the option of choosing fact tables and marking existing materialized views to retain.

Click To expand
Figure 4.10: Summary Advisor Wizard—filters.

Once the analysis is completed, the recommendations are displayed graphically, as shown in Figure 4.11. The X-axis of the graph shows the space used, and the Y-axis shows the cumulative performance gained from the recommendations. Users can decide how much space they would actually like to use by moving the arrow.

Click To expand
Figure 4.11: Summary Advisor Wizard—recommendation graph.

The recommendations that fit the chosen space budget can be viewed in detail by clicking the View/Modify Recommendations button. Figure 4.12 shows the wizard page for the recommendations. The user can change the materialized view name and options for any newly recommended materialized views.

Click To expand
Figure 4.12: Summary Advisor Wizard—view/modify recommendations.

Once the required modifications are made, you can press the Next button in Figure 4.11 to see the SQL required to implement the recommendations, as shown in Figure 4.13. You can also save the generated SQL statements to a file. Pressing the Finish button will implement the recommendations. In order to implement the recommendations, you must have the requisite privileges.

Click To expand
Figure 4.13: Summary Advisor Wizard—implement recommendations.

Next, we will see how to run the summary advisor on the command line.

4.7.2 Using the DBMS_OLAP package

The Summary Advisor can be run without the graphical interface using the procedures in the DBMS_OLAP package. We will show a typical session using the Summary Advisor APIs.

Loading workload

The first step to run the Summary Advisor is to load the workload. The workload can be loaded from the SQL cache, a user-defined table, or generated using the dimension and constraint information in the database. In the following example, we load the workload from the SQL cache using the LOAD_WORKLOAD_CACHE procedure. Before issuing the API to load the workload, you must create an identifier for the workload using the CREATE_ID procedure. This will later be provided to the RECOMMEND_MVIEW_STRATEGY procedure.

Variable workload_id number;
EXECUTE DBMS_OLAP.CREATE_ID(:workload_id);

EXECUTE DBMS_OLAP.LOAD_WORKLOAD_CACHE (:workload_id,
                                       DBMS_OLAP.WORKLOAD_NEW,
                                       DBMS_OLAP.FILTER_NONE,
                                       'Easydw', 1);

User-defined workload

Sometimes you may have a specific set of SQL statements you wish to tune. The SQL cache may not accurately represent these statements. In this case, you can place your statements in a table and use the LOAD_WORKLOAD_USER procedure to provide this workload to the Summary Advisor. The user workload table must at the minimum specify the query text and the user executing the query. To get better recommendations, you should provide some statistics, such as frequency and response time, if available.

The following example shows the format of a user workload table and how it is loaded.

CREATE TABLE my_workload
(query           varchar2(2000) not null,
 owner           varchar2(30)   not null,
 application     varchar2(30),
 frequency       number,
 lastuse         number,
 priority        number,
 responsetime    number,
 resultsize      number,
 sql_addr        number,
 sql_hash        number
);

<insert workload statements into my_workload>

variable workload_id number;

execute dbms_olap.create_id (:workload_id);

execute DBMS_OLAP.LOAD_WORKLOAD_USER (:workload_id,
                                      DBMS_OLAP.WORKLOAD_NEW,
                                      DBMS_OLAP.FILTER_NONE,
                                      'EASYDW', 'MY_WORKLOAD');

Once the workload is loaded, you follow the remaining steps, discussed in the following text, to obtain recommendations.

Filters

As discussed earlier, filters can be used to reduce the number of queries to be analyzed. Filters can be based on tables or schemas referenced in the queries, frequency, response time, cardinality of tables, or a user-defined priority field (for user-defined workload). In the following example, we will create a filter that only picks queries that have EASYDW tables in them. This narrows the focus of the Advisor, thereby reducing its analysis time. Further, we will only choose queries that occur at least ten times. As with the workload, you must create an identifier for the filter using the CREATE_ID procedure. We will use this identifier to provide the filter to the recommendation procedure.

Variable filter_id number;
EXECUTE DBMS_OLAP.CREATE_ID(:filter_id);

EXECUTE DBMS_OLAP.ADD_FILTER_ITEM(:filter_id, 'SCHEMA',  'EASYDW',
                                   NULL, NULL, NULL, NULL);
EXECUTE  DBMS_OLAP.ADD_FILTER_ITEM (:filter_id,  'FREQUENCY',
                                   NULL, 10, NULL, NULL, NULL);

Recommending materialized views

Now we are ready to start the recommendation process. To generate recommendations, we use the DBMS_OLAP.RECOMMEND_MVIEW_STRATEGY procedure. Before running this procedure, you must generate a run_id using the CREATE_ID procedure. This is used to identify the results of the Advisor's analysis in the SYSTEM.MVIEW_RECOMMENDATIONS catalog view.

The RECOMMEND_MVIEW_STRATEGY procedure must be provided with the workload and filters using the identifiers—workload_id and filter_id, that we had generated earlier. Optionally, you can also provide a storage budget available for materialized views (500 MB in the following example) and a list of materialized views that should not be dropped.

Variable run_id number;
EXECUTE DBMS_OLAP.CREATE_ID(:run_id);

EXECUTE DBMS_OLAP.RECOMMEND_MVIEW_STRATEGY(:run_id, :workload_id,
                                           :filter_id, 500000, 100,
                                           NULL, NULL);

Summary advisor reports and scripts

Once the recommendations have been generated, you can generate a report or script using the procedures DBMS_OLAP.GENERATE_MVIEW_REPORT and GENERATE_MVIEW_SCRIPT, respectively. The script provides the SQL statements used to generate the recommended materialized views.

In order to generate script and report files, the user issuing these procedures must have been granted read and write permissions on the specific directories. To grant user EASYDW read and write access to the h:\oracle\reports directory you would issue the following statement:

EXECUTE DBMS_JAVA.GRANT_PERMISSION('EASYDW',
                                   'java.io.FilePermission',
                                   'h:\oracle\reports\*',
                                   'read, write');

The following code segment is a script generated by the Summary Advisor for the previous example. We have specified that the script should use the SUMMARY tablespace for the generated materialized view statements. The script is placed in the directory that was granted permissions previously.

EXECUTE DBMS_OLAP.GENERATE_MVIEW_SCRIPT('h:\oracle\reports\mview_rec.sql',
                                :run_id, 'SUMMARY');

/*****************************************************************************
**  Oracle Summary Advisor 9i - Production
**
**  Summary Advisor Recommendation Script
*****************************************************************************/

/*****************************************************************************
**  Recommendations for run ID #59
*****************************************************************************/

/*****************************************************************************
**  Rank 1
**  Storage 16,240 bytes
**  Gain 78.72%
**  Benefit Ratio 201,344.00
**
**  SELECT easydw.customer.customer_id, easydw.time.month, easydw.time.year,
**         sum(easydw.purchases.purchase_price), count(
**         easydw.purchases.purchase_price), count(*)
**      FROM easydw.purchases, easydw.time, easydw.customer
**      WHERE easydw.time.time_key = easydw.purchases.time_key
**        AND easydw.customer.customer_id = easydw.purchases.customer_id
**      GROUP BY easydw.customer.customer_id, easydw.time.month,
**               easydw.time.year
*****************************************************************************/

CREATE MATERIALIZED VIEW mv_id_59_rank_1
  TABLESPACE summary
  BUILD IMMEDIATE
  REFRESH COMPLETE
  ENABLE QUERY REWRITE AS
    SELECT EASYDW.CUSTOMER.CUSTOMER_ID, EASYDW.TIME.MONTH, EASYDW.TIME.YEAR,
SUM("EASYDW"."PURCHASES"."PURCHASE_PRICE"),
COUNT("EASYDW"."PURCHASES"."PURCHASE_PRICE"), COUNT(*) FROM EASYDW.PURCHASES,
EASYDW.TIME,
EASYDW.CUSTOMER WHERE EASYDW.TIME.TIME_KEY = EASYDW.PURCHASES.TIME_KEY AND
EASYDW.CUSTOMER.CUSTOMER_ID
= EASYDW.PURCHASES.CUSTOMER_ID GROUP BY EASYDW.CUSTOMER.CUSTOMER_ID,
EASYDW.TIME.MONTH,
EASYDW.TIME.YEAR;

<remainder of script skipped>

You can also get a detailed HTML report of the Summary Advisor analysis by issuing the DBMS_OLAP.GENERATE_MVIEW_REPORT procedure, as shown in the following example. You can customize the report to include different detailed sections of the report, such as workload details, filters, recommendations, and existing materialized view use. If the run_id is not passed in, a report is generated for all data for the user issuing the procedure. Figure 4.14 shows a section of a Summary Advisor report displayed in a Web browser.

Click To expand
Figure 4.14: Summary Advisor report.
EXECUTE DBMS_OLAP.GENERATE_MVIEW_REPORT
        ('h:\oracle\reports\report_rec.html',
          :run_id, DBMS_OLAP.RPT_ALL);

Hypothetical workload

If you are still in the process of developing the application, you may not have the set of SQL statements that may get executed. The Summary Advisor can still be used if you have the schema for the queries and some additional information such as constraints or dimensions defined. In this case, the Summary Advisor will determine a hypothetical workload using the relationships between various columns and then recommend materialized views for it. This works well for star schemas, which are typical in a data warehouse. To generate recommendations using a hypothetical workload, pass in a workload_id of DBMS_OLAP.WORKLOAD_NONE in the DBMS_OLAP.RECOMMEND_MV procedure.

EXECUTE DBMS_OLAP.RECOMMEND_MVIEW_STRATEGY(:run_id,
                                           DBMS_OLAP.WORKLOAD_NONE,
                                           DBMS_OLAP.FILTER_NONE,
                                           500000, 100, NULL, NULL);

You can view the recommendations and generate reports and scripts as described previously.

Purging results

You can clean up the results of the analysis by calling the DBMS_OLAP.PURGE_RESULTS procedure.

EXECUTE DBMS_OLAP.PURGE_RESULTS(:run_id);

You can also use DBMS_OLAP.RUNID_ALL to purge all Advisor results.

EXECUTE DBMS_OLAP.PURGE_RESULTS(DBMS_OLAP.RUNID_ALL);

You can remove a workload that is no longer required by using the PURGE_WORKLOAD procedure by passing in the specific workload_id or DBMS_OLAP.WORKLOAD_ALL to remove all workloads. Similarly, you can remove a filter using the PURGE_FILTER procedure.

Monitoring existing materialized view use

You may sometimes want to know which summaries are being used. A major problem for a DBA is to determine when it is safe to remove a summary from the data warehouse. The DBMS_OLAP package provides the EVALUATE_MVIEW_STRATEGY procedure to evaluate the utilization of existing materialized views. This helps determine if some materialized views are not being used by your current workload. The results of the evaluation may be viewed in the SYSTEM.MVIEW_EVALUATIONS view.

EXECUTE DBMS_OLAP.EVALUATE_MVIEW_STRATEGY(:run_id, :workload_id,
                                          :filter_id);

SELECT runid, mview_owner, mview_name, storage, freq,
       cumulative_benefit as benefit
FROM SYSTEM.MVIEW_EVALUATIONS;

RUNID MVIEW_OWNER  MVIEW_NAME          STORAGE  FREQ    BENEFIT
----- -----------  ------------------- -------  ----   ---------
141   EASYDW       SALES_MV              1539    12     41351428
141   EASYDW       PRODUCT_SALES_BY_MON 35303     5      4295396

Estimating the size of a materialized view

If you were going to create a summary, it would be nice to know how much space it is likely to occupy. You can find out this information by calling the procedure DBMS_OLAP.ESTIMATE_SUMMARY_SIZE, as shown in the following code segment. This procedure is very easy to use: Simply pass in the SELECT expression used to define the materialized view.

SET SERVEROUTPUT on

DECLARE
no_of_rows NUMBER;
mv_size NUMBER;
BEGIN
no_of_rows :=0;
mv_size :=0;

DBMS_OLAP.estimate_summary_size ('purchases_by_county',
 'SELECT SUM(ps.purchase_price), ct.county
  FROM PURCHASES ps, CUSTOMER ct
  WHERE ps.customer_id = ct.customer_id
  GROUP BY ct.county' , no_of_rows, mv_size);

DBMS_OUTPUT.put_line ( 'SELECT on Purchases GROUP by County ');
DBMS_OUTPUT.put_line ( 'No of Rows: ' || no_of_rows );
DBMS_OUTPUT.put_line ( 'Size of Summary (bytes): ' || mv_size);
DBMS_OUTPUT.put_line ( '');
END;
/

SELECT on Purchases GROUP by County
No of Rows: 3
Size of Materialized view (bytes): 96

In this example, we report the number of rows and the expected materialized view size. Also note that in this example, we have used DBMS_OUTPUT.PUT_LINE to display the results from the package.


Summary

4.8 Summary

Summary Management in Oracle 9i provides a very powerful set of tools to improve query performance in your warehouse. Materialized views can be used to precompute the results of frequently asked queries. With query rewrite, the queries will be transparently rewritten to use the materialized views. Fast refresh and partition change tracking allow the summary tables to be kept up-to-date when your warehouse gets loaded. Dimensions allow you to represent hierarchical information about your data to take advantage of rollup relationships during query rewrite. The Summary Advisor can be used to recommend the right materialized views for your application.

In the next chapter, we will discuss in detail the various activities involved in loading new data into the warehouse.


Chapter 5: Loading Data into the Warehouse

5.1 The ETL process

Populating a data warehouse involves all of the tasks related to getting the data from the source operational systems, cleansing and transforming the data to the right format and level of detail, loading the data into the target data warehouse, and preparing the data for analysis purposes.

Figure 5.1 shows the steps making up the extraction, transformation, and load (ETL) process. Data is extracted from the source operational systems and transported to the staging area. The staging area is a temporary holding place used to prepare the data. The staging area may be a set of flat files, temporary staging tables in the Oracle warehouse, or both. The data is integrated with other data, cleansed, and transformed into a common representation. The data is then loaded into the target data warehouse tables. Sometimes this process is referred to as ETT, extraction: transportation, and transformation.

Click To expand
Figure 5.1: The ETL process.

During the initial population of the data warehouse, historical data is loaded. The data in the operational systems may often be in multiple formats. If, for instance, the point-of-sales operational system was replaced two years ago, the current two years of history will be in one format, while data older than two years will be in another format.

After the initial historical load, new transaction and event data needs to be loaded on a periodic basis. This is typically done on a regular time schedule, such as at the end of the day, week, or month. During the load and while the indexes and materialized views are being refreshed, the data is generally unavailable to warehouse users for querying. The period of time allowed for inserting the new data is called the "batch window." The batch window is a continuously shrinking amount of time, as more businesses are on line for longer periods of time. Higher availability can be achieved by partitioning the fact table. While data is loaded into a new partition, or being updated in an existing partition, the rest of the partitions are still available for use.

A large portion of the work in building a data warehouse will be devoted to the ETL process. Finding the data from the operational systems, creating extraction processes to get the data out, transporting, filtering, cleansing, transforming, integrating data from multiple sources, and loading the data into the warehouse all can take a considerable amount of time.

If you are not already familiar with the company's data, part of the difficulty in developing the ETL process is gaining an understanding of the data. One object can have different names in different systems. Even worse, two different things could have the same name. It can be a challenge to discover all of this, particularly in a company where the systems are not well documented. Each column in the target data warehouse must be mapped to the corresponding column in the source system. Some of the data will be mapped directly; other data will need to be derived and transformed into a different format.

Once the data is loaded into the warehouse, further processing to integrate it with existing data, update indexes, gather statistics, and refresh materialized views needs to take place prior to it being "published" as ready for users to access. Once the data is published, it should not be updated again until the next batch window, to ensure that users do not receive different answers to the same query asked at a different time.


Extracting data from the operational systems

5.2 Extracting data from the operational systems

Once you have identified the data you need in the warehouse for analysis purposes, you need to locate the operational systems within the company that contain that data. The data needed for the warehouse is extracted from the source operational systems and written to the staging area, where it will later be transformed. To minimize the performance impact on the source database, data is generally downloaded without applying any transformations to it.

Often the owners of the operational systems will not allow the warehouse developers direct access to those systems but will provide periodic extracts. These extracts are generally in the form of flat, sequential operating system files, which will make up the staging area.

Application programs need to be developed to select the fields and records needed for the warehouse. If the data is stored in a legacy system, these may be written in COBOL and require special logic to handle things such as repeating fields in the "COBOL occurs clause." The data warehouse designers need to work closely with the application developers for the OLTP systems who are building the extract scripts to provide the necessary columns and formats of the data.

As part of designing the ETL process, you need to determine how frequently data should be extracted from the operational systems. It may be at the end of some time period or business event, such as at the end of the day or week or upon closing of the fiscal quarter. It should be clearly defined what is meant by the "end of the day" or the "last day of the week," particularly if you have a system used across different time zones. The extraction may be done at different times for different systems and staged to be loaded into the warehouse during an upcoming batch window. Another aspect of the warehouse design process involves deciding what level of aggregation is to be used to answer the business queries. This also has an impact on what and how much data is extracted and transported across the network.

Some operational systems may be in relational databases, such as Oracle 8i or 9i, Oracle Rdb, DB2/MVS, Microsoft SQL Server, Sybase, or Informix. Others may be in a legacy database format, such as IMS or Oracle DBMS. Others may be in VSAM, RMS indexed files, or some other structured file system.

If you are able to access the source systems directly, you can get the data out by a variety of techniques depending on the type of system the data is in. For small amounts of data, you can use a gateway or ODBC. For larger amounts of data, a custom program directly connecting to the source database in the database's native Application Programming Interface (API) can be written. Many ETL tools simplify the extraction process by providing connectivity to the source.

5.2.1 Identifying data that has changed

After the initial load of the warehouse, as the source data changes, the data in the warehouse must be updated or refreshed to reflect those changes on a regular basis. A mechanism needs to be put into place to monitor and capture changes of interest from the operational systems. Rather than rebuilding the entire warehouse periodically, only the changes need to be applied. By isolating changes as part of the extraction process, less data needs to be moved across the network and loaded into the data warehouse.

Changed data includes both new data that has been added as well as updates and deletes to existing data. For example, in the EASYDW warehouse, we are interested in all new orders as well as updates to existing product information and customers. If we are no longer selling a product, the product is deleted from the order-entry system, but we still want to retain the history in the warehouse. This is why surrogate keys are recommended for use in the data warehouse. If the product_key is reused in the production system, it does not affect the data warehouse records.

In the data warehouse, it is not uncommon to change the dimension tables, because a column such as a product description may change. Part of the warehouse design involves deciding how changes to the dimensions will be reflected. If you need to keep one version of the old product description, you could have an additional column in the table to store both the current description and the previous description. If you needed to keep all the old product descriptions, you would have to create a new row for each change, assigning different key values. In general, you should try to avoid updates to the fact table.

There are various ways to identify the new data. One technique to determine the changes is to include a time stamp to record when each row in the operational system was changed. The data extraction program then selects the source data based on the time stamp of the transaction and extracts all rows that have been updated since the time of the last extraction. When moving orders from the order processing system into the EASYDW warehouse, this technique can be used by selecting rows based on the purchase_date, as illustrated later in this chapter.

If the source is a relational database, triggers can be used to identify the changed rows. Triggers are stored procedures that can be invoked before or after an event, such as when an insert, update, or delete occurs. The trigger can be used to save the changed records in a table from where the extract process can later retrieve the changed rows. Be very careful of triggers in high-volume applications, since they can add significant overhead to the operational system.

Sometimes you may not be able to change the schema to add a time stamp or trigger. Your system may already be heavily loaded, and you do not want to degrade the performance in any way. Or the source may be a legacy system, which does not have triggers. Therefore, you may need to use a file comparison to identify changes. This involves keeping before and after images of the extract files to find the changes. For example, you may need to compare the recent extract with the current product or customer list to identify the changes.

Changes to the metadata or data definitions must also be identified. Changes to the structure of the operational system, such as adding or dropping a column, impact the extraction and load programs, which may need to be modified to account for the change.

5.2.2 Oracle change data capture

In Oracle 9i a new technique was introduced to facilitate identifying changes when the source system is an Oracle 9i database. The results of all INSERT, UPDATE, and DELETE operations can be saved in tables called change tables. The data extraction programs can then select the data from the change tables.

Change data capture, often referred to as CDC, uses a publish-subscribe interface to capture and distribute the change data, as illustrated in Figure 5.2. The publisher, usually a DBA, determines which user tables in the operational system are used to load the warehouse and sets up the system to capture and publish the change data. A change table is created for each source table with data that needs to be moved to the warehouse.

Click To expand
Figure 5.2: Change data capture publish/ subscribe architecture.

The extract programs then subscribe to the source tables; therefore, there can be any number of subscribers. Each subscriber is given his or her own view of the change table. This isolates the subscribers from each other while they are simultaneously accessing the same change tables. The subscribers use SQL to select the change data from their subscriber views. They see just the columns they are interested in and only the rows that they have not yet processed. If the updates of a set of tables are dependent on each other, the change tables can be grouped into a change set. If, for example, you had an order header and an order detail table, these two tables would be grouped together in the same change set to maintain transactional consistency.

In Oracle 9i the changes are captured synchronously, in real time, as part of the transaction on the operational system. The change data is generated as DML operations are performed on the source tables. When a new row is inserted into the user table, it is also stored in the change table. When a row is updated in a user table, the updated columns are stored in the change table. The old values, new values, or both can be written to the change table. When a row is deleted from a user table, the deleted row is also stored in the change table. This does add overhead to each transaction, but simplifies the data extraction process.

Publishing change data

In the EASYDW warehouse, we are interested in all new orders from the order-entry system. The DBA creates the change tables, using the DBMS_LOGMINER_CDC_PUBLISH.CREATE_CHANGE_TABLE supplied procedure, and specifies a list of columns that should be included. A change table contains changes from only one source table.

In many of the examples in this chapter we will use a schema named OLTP, which is part of the operational database. In the following example, the DBA uses the CREATE_CHANGE_TABLE supplied procedure to capture the PRODUCT_ID, CUSTOMER_ID, PURCHASE_DATE, PURCHASE_TIME, PURCHASE_PRICE, SHIPPING_CHARGE, and TODAY_SPECIAL_OFFER columns from the ORDERS table in the OLTP schema. The name of the change table is ORDERS_CHANGE_TABLE. Since there is only one change table in our example, the default and only supported set type in Oracle 9i, SYNC_SET, is used.

After the column list there are a number of parameters, which allow you to specify:

  • Whether you want the change table to contain the old values for the row, the new values, or both

  • Whether you want a row sequence number, which provides the sequence of operations within a transaction

  • The row ID of the changed row

  • The user who changed the row

  • The time stamp of the change

  • The object id of the change record

  • A source column map, which indicates what source columns have been modified

  • A target column map to track which columns in the change table have been modified

  • An options column to append to a CREATE TABLE DDL statement

An application can check either the source column map or the target column map to determine which columns have been modified.

SQL> EXECUTE DBMS_LOGMNR_CDC_PUBLISH.CREATE_CHANGE_TABLE -
('oltp', 'orders_change_table', 'SYNC_SET', 'oltp', 'orders', -
'product_id VARCHAR2(8), -
 customer_id VARCHAR2(10),-
 purchase_date date, -
 purchase_time NUMBER(4,0), -
 purchase_price NUMBER(6,2), -
 shipping_charge NUMBER(5,2), -
 today_special_offer VARCHAR2(1)', -
 'BOTH', 'Y','N','Y','Y','N','Y','Y',NULL);

A sample of the output will be seen later in the chapter. To see a list of change tables that have been published, query the CHANGE_TABLES dictionary table.

SQL> SELECT CHANGE_TABLE_NAME FROM CHANGE_TABLES;

CHANGE_TABLE_NAME
------------------
ORDERS_CHANGE_TABLE

The DBA then grants SELECT privileges on the change table to the subscribers.

Subscribing to change data

The extraction programs create subscriptions to access the change tables. A subscription can contain data from one or more change tables in the same change set.

The all_source_tables dictionary view lists the source tables that have already been published by the DBA. In this example, changes for the ORDERS table in the OLTP schema have been published.

SQL> SELECT * FROM ALL_SOURCE_TABLES;

SOURCE_SCHEMA_NAME             SOURCE_TABLE_NAME
------------------------------ ------------------
OLTP                           ORDERS

Creating a subscription

There are several steps to creating a subscription, as follows:

  1. Get a subscription handle.

  2. List all the tables and columns the extract program wants to subscribe to.

  3. Activate the subscription.

In order to create a subscription to the change data, obtain a subscrip-tion handle to the change set using the GET_SUBSCRIPTION_HANDLE procedure. The subscription handle will be passed into the other supplied procedures. In the following example, a variable called "subhandle" is first created.

SQL> VARIABLE subhandle NUMBER;
SQL> EXECUTE DBMS_LOGMNR_CDC_SUBSCRIBE.GET_SUBSCRIPTION_HANDLE -
(CHANGE_SET => 'SYNC_SET', -
 DESCRIPTION => 'Changes to orders table', -
 SUBSCRIPTION_HANDLE => :subhandle);

Next, a subscription is created using the SUBSCRIBE procedure. A subscription can contain one or more tables from the same change set. The SUBSCRIBE procedure lists the schema, table, and columns of change data that the extract program will use to load the warehouse. In this example, the subscribe procedure is used to get changes from all the columns in the ORDERS table in the OLTP schema. The subscribe procedure is executed once for each table in the subscription. In this example we were only interested in changes from one table.

SQL> EXECUTE DBMS_LOGMNR_CDC_SUBSCRIBE.SUBSCRIBE -
(SUBSCRIPTION_HANDLE => :subhandle, -
 SOURCE_SCHEMA => 'oltp', -
 SOURCE_TABLE => 'orders', -
 COLUMN_LIST => 'product_id,customer_id,purchase_date, -
 purchase_time,purchase_price,shipping_charge,today_special_offer');

After subscribing to all the change tables, the subscription is activated using the ACTIVATE_SUBSCRIPTION procedure. Activating a subscription is done to indicate that all tables have been added, and the subscription is now complete.

SQL> EXECUTE DBMS_LOGMNR_CDC_SUBSCRIBE.ACTIVATE_SUBSCRIPTION -
 (SUBSCRIPTION_HANDLE => :subhandle);

Once a subscription has been created, and new data added to the source tables, it is available for processing.

Processing the change data

To illustrate how change data is processed, two rows are inserted into the orders table. As data is inserted into the orders table, the changes are also stored in the ORDERS_CHANGE_TABLE.

SQL> INSERT INTO oltp.orders(order_id,product_id, -
customer_id, purchase_date, purchase_time, purchase_price, -
shipping_charge,today_special_offer -
sales_person_id,payment_method) -
VALUES ('123','SP1031','AB123495','01-JAN-02',
1031,156.45,6.95,'N','SMITH','VISA');

1 row created.

SQL> INSERT INTO oltp.orders(order_id,product_id, -
customer_id, purchase_date, purchase_time, purchase_price, -
shipping_charge,today_special_offer -
sales_person_id,payment_method) -
VALUES ('123','SP1031','AB123495','01-FEB-02',
1031,156.45,6.95,'N','SMITH','VISA');

1 row created.

In order to process the change data, a program loops through the steps listed in the following text and illustrated in Figure 5.3. A change table is dynamic; new change data is appended to the change table at the same time the extraction programs are reading from it. In order to present a consistent view of the contents of the change table, data is viewed for a window of time. Prior to accessing the data, the window is extended. In Figure 5.3 rows 1-8 are available in the first window. While the program was processing these rows, rows 9-15 were added to the change table. Purging the first window, and extending the window again can access rows 9-15.

Click To expand
Figure 5.3: Querying change data.

Rather than accessing the change table directly, the subscribers need to create a view. The program prepares a subscriber view, selects the data from the change table using the view, and then drops the view.

Step 1: Extend the window

Change data is only available for a window of time: from the time the EXTEND_WINDOW procedure is invoked until the PURGE_WINDOW procedure is invoked. To see new data added to the change table, the window must be extended using the EXTEND_WINDOW procedure.

SQL> EXECUTE DBMS_LOGMNR_CDC_SUBSCRIBE.EXTEND_WINDOW -
(SUBSCRIPTION_HANDLE => :subhandle);
Step 2: Prepare a subscriber view

Instead of accessing the change tables directly, the subscribers need to create a view to retrieve the data. In the following example, a variable called "viewname" is first created.

SQL> VARIABLE viewname VARCHAR2(4000);

The view is created using the PREPARE_SUBSCRIBER_VIEW procedure. The contents of the view will not change, even when more data is added to the change table.

SQL> EXECUTE DBMS_LOGMNR_CDC_SUBSCRIBE.PREPARE_SUBSCRIBER_VIEW -
(SUBSCRIPTION_HANDLE => :subhandle, -
SOURCE_SCHEMA => 'oltp', -
SOURCE_TABLE => 'orders', -
VIEW_NAME => :viewname);
Step 3: Select data from the view

In this example, the contents of the view will be examined. First the name of the view is determined and then used as part of the SELECT statement.

SQL> PRINT viewname;

VIEWNAME
--------------
CDC#CV$2131489

The first column of the output shows the operation: "I" for insert. Next the commit scn and commit time are listed, followed by the user who made the change. The new data is listed for each row. The row source ID, indicated by RSID$, shows the order of the statements in the transaction.

SQL> SELECT OPERATION$, CSCN$, COMMIT_TIMESTAMP$,
     USERNAME$, CUSTOMER_ID, PRODUCT_ID, RSID$
     FROM cdc#cv$2131489;

OP  CSCN$ COMMIT_TI USERNAME$ CUSTOMER_ID PRODUCT_ID RSID$
--- ----- --------- --------- ----------- ---------- ----
I      345643 28-FEB-02   OLTP    AB123495   SP1031     1
I      345643 28-FEB-02   OLTP    AB123495   SP1031     2
Step 4: Drop the subscriber view

When you are through processing the data in the subscriber view, it is dropped using the drop_subscriber_view procedure.

SQL> EXECUTE DBMS_LOGMNR_CDC_SUBSCRIBE.DROP_SUBSCRIBER_VIEW -
    (SUBSCRIPTION_HANDLE => :subhandle, -
     SOURCE_SCHEMA => 'oltp', -
     SOURCE_TABLE => 'orders');
Step 5: Purge the window

The window is purged when the data is no longer needed using the purge_window procedure. When all subscribers have purged their windows, the data in those windows is automatically deleted.

SQL> EXECUTE DBMS_LOGMNR_CDC_SUBSCRIBE.PURGE_WINDOW -
    (SUBSCRIPTION_HANDLE => :subhandle);

Ending the subscription

When an extract program is no longer needed, you can end the subscription, using the drop_subscription procedure.

SQL> EXECUTE DBMS_LOGMNR_CDC_SUBSCRIBE.DROP_SUBSCRIPTION -
    (SUBSCRIPTION_HANDLE => :subhandle);

Transporting the changes to the staging area

Now that the changes have been captured from the operational system, they need to be transported to the staging area. The extract program could write them to a data file outside the database, use FTP to copy it, and use SQL*Loader or external tables to load the change data into the staging area. Alternatively, the changes could be written to a table and moved to the staging area using transportable tablespaces. Both these techniques are discussed later in this chapter.


Transforming the data into a common representation

5.3 Transforming the data into a common representation

Once the data has been extracted from the operational systems, it is ready to be cleansed and transformed into a common representation. Differences between naming conventions, storage formats, data types, and encoding schemes must all be resolved. Duplicates are removed, relationships are validated, and unique key identifiers are added. In this section, various types of transformations will be introduced; later in the chapter we'll see specific examples of transformations.

5.3.1 Integrating data from multiple sources

Often the information needed to create a table in the warehouse comes from multiple source systems. If there is a field in common between the systems, the data can be joined via that column.

Integrating data from multiple sources can be very challenging. Different people may have designed the operational systems, at different times, using different styles, standards, and methodologies. They may use different technology (e.g., hardware platforms, database management systems, and operating system software). If data is coming from an IBM mainframe, the data may need to be converted from EBCDIC to ASCII or from big endian to little endian or vice versa.

To compound the problem, there may not be a common identifier in the source systems. For example, when creating the customer dimension, there may not be a CUSTOMER_ID in each system. You may have to look at customer names and addresses to determine that it is the same customer. These may have different spacing, case, and punctuation. Oracle Warehouse Builder helps address the customer deduplication problem.

5.3.2 Cleansing data

The majority of operational systems contain some dirty data, which means that there may be:

  • Duplicate records

  • Data missing

  • Data containing invalid values

  • Data pointing to primary keys that do not exist

Sometimes business rules are enforced by the applications; other times, by integrity constraints within the database; and sometimes there may be no enforcement at all.

Data must be standardized. For example, any given street address, such as 1741 Coleman Ave., can be represented in many ways. The word "Avenue" may be stored as "Ave," "Ave.," "Avenue," or "AVE." Search and replace transforms allow you to search for any of these values and replace them with the standard value you've chosen for your warehouse.

You may want to check the validity of certain types of data. If a product is sold only in three colors, you can validate that the data conforms to a list of values, such as red, yellow, and green. This list may change over time to include new values-for example, in February 2002 the product may also be available in blue. You may want to validate data against a larger list of values stored in a table, such as the states within the United States.

Some types of cleansing involve combining and separating character data. You may need to concatenate two string columns-for example, combining LAST_NAME, comma, and FIRST_NAME into the CUSTOMER_NAME column. Or you may need to use a substring operation to divide a string into separate parts, such as separating the area code from a phone number.

An important data integrity step involves enforcement of one-to-one and one-to-many relationships. Often these are checked as part of the transformation process rather than by using referential integrity constraints in the warehouse.

5.3.3 Deriving new data

While loading the data, you may want to perform calculations or derive new data from the existing data. For example, you may want to keep a running total or count of records as they are moved from the source to the target database.

During the design process, the appropriate level of granularity for the warehouse is determined. It is often best to store data at various levels of granularity with different retention and archive periods. The most fine-grained transaction data will usually be retained for a much shorter period of time than data aggregated at a higher level. Transaction granular sales data is necessary to analyze which products are purchased together. Daily sales of a product by store are used to analyze regional trends and product performance.

Data may be aggregated as part of the transformation process. If you did not want to store the detailed transactions in your data warehouse, the data can be aggregated prior to moving it to the data warehouse.

5.3.4 Generating warehouse keys

Instead of using the keys that were used in the operational system, a common design technique is to make up a new key, called the surrogate or synthetic key, to use in the warehouse. The surrogate key is usually a generated sequence of integers.

Surrogate keys are used for a variety of reasons. The keys used in the operational system may be long character strings, with meaning embedded into the components of the key. Because surrogate keys are integers, the fact tables and B*tree indexes are smaller, with fewer levels, and take less space, improving query response time. Surrogate keys provide a degree of isolation from changes in the operational system.

If the operational system changes the product-code naming conventions, or format, all data in the warehouse does not have to be changed. When one company acquires another, you may need to load products from the newly acquired company into the warehouse. It is highly unlikely that both companies used the same product encoding schemes. If there is a chance that the two companies used the same product key for different products, then the product key in the warehouse may need to be extended to add the company ID as well. The use of surrogate keys can greatly help integrate the data in these types of situations.

Both the surrogate keys and operational system keys are stored in the dimension table, as shown in Figure 5.4, where product code SR125 is known in the data warehouse as PRODUCT_ID 1. Therefore, we can see that in the fact table, the product key is stored as 1. However, users can happily query using code SR125, completely unaware of the transformation being done within the data warehouse.

Click To expand
Figure 5.4: The use of surrogate keys in the warehouse.

The surrogate key is used in the fact table as the column that joins the fact table to the dimension table. In this example, there are two different formats for product codes. Some are numeric, separated by a dash: 123-118. Others are a mix of alphanumeric and numeric characters: SR125. As part of the ETL process, as each fact record is loaded, the surrogate key is looked up from the dimension tables and stored in the fact table.

5.3.5 Choosing the optimal place to perform the transformations

Transformations of the data may be done at any step in the ETL process. You need to decide the most efficient place to do each transformation: at the source, in the staging area, during the load operation, or in temporary tables once the data is loaded into the warehouse. Several new features have been added to Oracle 9i to facilitate performing transformations.

  • Transformations can be done as part of the extraction process. In general, it is best to do filtering types of transformations whenever possible at the source. This allows you to select only the records of interest for loading into the warehouse. Ideally, you want to extract only the data that has been changed since your last extraction. While transformations could be done at the source operational system, an important consideration is to minimize the additional load the extraction process puts on the operational system.

  • Transformations can be done in a staging area prior to loading the data into the data warehouse. When data needs to be integrated from multiple systems, this cannot be done as part of the extraction process. You can use flat files as your staging area, your Oracle database as your staging area, or a combination of both. If your incoming data is in a flat file, it is probably more efficient to finish your staging processes prior to loading the data into the Oracle warehouse. Transformations that require sorting, sequential processing, and row-at-a-time operations can be done efficiently in the flat file staging area.

    If your source is an Oracle 8i or 9i database, transportable tablespaces make it easy to move data into the warehouse without first extracting the data into an external table. In this case, it makes sense to do the transformations in temporary staging tables once the data is in the warehouse. By doing transformations in Oracle, if the data can be processed in bulk using SQL set operations, they can be done in parallel. If some of the data needed for the transformation purposes are already in the warehouse, it may be more efficient to do the transformation in the warehouse. For example, key lookups to obtain the surrogate key from the dimension table can be efficiently performed in staging tables in the database. Section 5.5 describes some examples of performing transformations in a staging area within the Oracle database.

  • Transformations may be done during the load process. Some simple types of transformations can be done as the data is being loaded using SQL*Loader. For example, it can be used to change the case of a character column to uppercase. This is best done when a small number of rows need to be added-for example, when initially loading a dimension table. In Oracle 9i, a new feature-external tables-was introduced to facilitate more complex transformations of the data as part of the load process. The following text describes some examples of transformations while loading the data into the warehouse.


Loading the warehouse

5.4 Loading the warehouse

When loading the warehouse, the dimension tables are generally loaded first. The dimension tables contain the surrogate keys or other descriptive information needed by the fact tables. When loading the fact tables, information is looked up from the dimension tables and added to the columns in the fact table.

When loading the dimension table, you need both to add new rows and make changes to existing rows. For example, a customer dimension may contain tens of thousands of customers. Usually, only 10 percent or less of the customer information changes. You will be adding new customers and sometimes modifying the information about existing customers.

When adding new data to the dimension table, you need to determine if the record already exists. If it does not, you can add it to the table. If it does exist, there are various ways to handle the changes, based on whether you need to keep the old information in the warehouse for analysis purposes.

If a customer's address changes, there is generally no need to retain the old address, so the record can simply be updated. In a rapidly growing company the sales regions will change often. For example, "Canada" rolled up into the "rest of the world" until 1990 and then rolled up into the "Americas" after reorganization. If you needed to understand both the old geographic hierarchy as well as the new one, you can create a new dimension record containing all the old data plus the new hierarchy, giving the record a new surrogate key. Alternatively, you could create columns in the original record to hold both the previous and current values.

The time dimension contains one row for each unit of time that is of interest in the warehouse. In the EASYDW shopping example, purchases can be made on line 365 days a year, so every day is of interest. For each given date, information about the day is stored, including the day of the week, the week number, the month, the quarter, the year, and if it is a holiday. The time dimension may be loaded on a yearly basis.

When loading the fact table, you typically append new information to the end of the existing fact table. You do not want to alter the existing rows, because you want to preserve that data. For example, the PURCHASES fact table contains three months of data. New data from the source order-entry system is appended to the purchases fact table monthly. Partitioning the data by month facilitates this type of operation.

In the next sections we will take a look at different ways to load data, including:

  • SQL*Loader—inserts data into a new table or appends to an existing table when your data is in a flat file, external to the database

  • External tables—inserts data into a new table or appends to an existing table when your data is in a flat file, external to the database, and you want to transform this data while loading

  • Transportable tablespaces—used to move the data from between two Oracle databases, such as the operational system and the warehouse.

5.4.1 Using SQL*Loader to load the warehouse

One of the most popular tools for loading data is SQL*Loader, because it has been designed to load records as fast as possible. It can be used either from the operating system command line or via its wizard in Oracle Enterprise Manager (OEM), which we will discuss now.

Using Oracle Enterprise Manager Load Wizard

Figure 5.5 shows the Oracle Enterprise Manager Data Management Load Wizard, which can be used to help automate the scheduling of your SQL*Loader jobs. The wizard guides you through the process of loading data from an external file into the database according to a set of instructions in a control file. A batch job is submitted through Enterprise Manager to execute the load.

Click To expand
Figure 5.5: The Load Wizard.

You can launch the data management tools by selecting the database tools from the tools menu once the target database is selected in the navigator tree, as shown in Figure 5.6.

Click To expand
Figure 5.6: Accessing the Load Wizard.
The control file

The control file is a text file that describes the load operation. The role of the control file, which is illustrated in Figure 5.7, is to tell SQL*Loader which data file to load, how to interpret the records and columns, and into what tables to insert the data.

Click To expand
Figure 5.7: The control file.

The control file is written in SQL*Loader's data definition language. The following example shows the control file that would be used to add new product data into the product table in the EASYDW warehouse. The data is stored in the file product.dat. New rows will be appended to the existing table.

- Load product dimension
LOAD DATA
INFILE 'product.dat' append
INTO TABLE product
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY "'"
(product_id,
 product_name,
 category,
 cost_price,
 sell_price,
 weight,
 shipping_charge,
 manufacturer,
 supplier)
The data file

The following example shows a small sample of the data file product.dat. Each field is separated by a comma and optionally enclosed in a single quote. Each field in the input file is mapped to the corresponding column in the table. As the data is read, it is converted from the data type in the input file to the data type of the column in the database.

'SP1242', 'CD LX1','MUSC', 8.90, 15.67, 2.5, 2.95, 'RTG', 'CD Inc'
'SP1243', 'CD LX2','MUSC', 8.90, 15.67, 2.5, 2.95, 'RTG', 'CD Inc'
'SP1244', 'CD LX3','MUSC', 8.90, 15.67, 2.5, 2.95, 'RTG', 'CD Inc'
'SP1245', 'CD LX4','MUSC', 8.90, 15.67, 2.5, 2.95, 'RTG', 'CD Inc'

The data file is an example of data stored in stream format. A record separator, often a line feed or carriage return/line feed, terminates each record. A delimiter character, often a comma, separates each field. The fields may also be enclosed in single or double quotes.

In addition to the stream format, SQL*Loader supports fixed-length and variable-length format files. In a fixed-length file, each record is the same length. Normally each field in the record is also the same length. In the control file, the input record is described by specifying the starting position, length, and data type. In a variable-length file, each record may be a different length. The first field in each record is used to specify the length of that record.

SQL*Loader modes of operation

Figure 5.8 shows SQL*Loader's two modes of operation: conventional path and direct path.

Click To expand
Figure 5.8: SQL*Loader modes of operation.

Conventional path load should only be used to load small amounts of data, such as initially loading a small dimension table, or when loading data with data types not supported by direct path load, such as varrays. The conventional path load issues SQL INSERT statements. As each row is inserted, the indexes are updated, triggers are fired, and constraints are evaluated. When loading large amounts of data in a small batch window, direct path load can be used to optimize performance. Direct path load bypasses the SQL layer. It formats the data blocks directly and writes them to the database files. When running on a system with multiple processors, the load can be executed in parallel, which can result in significant performance gains.

Handling SQL*Loader errors

You can create additional files during the load operation to aid in the diagnosis and correction of any errors that may occur during the load. A log file is created to record the status of the load operation. This should always be reviewed to ensure the load was successful. Copies of the records that could not be loaded into the database because of data integrity violations can be saved in a "bad" file. This file can later be reentered once the data integrity problems have been corrected. If you receive an extract file with more records than you are interested in, you can load a subset of records from the file. The WHEN clause in the control file is used to select the records to load. Any records that are skipped are written to a discard file.

Select which optional files you would like created using the advanced option, as shown in Figure 5.9. In this example, we've selected a bad file and a log file.

Click To expand
Figure 5.9: SQL*Loader advanced options.
Scheduling the load operation

Enterprise Manager's job scheduling system allows you to create and manage jobs, schedule the jobs to run, and monitor progress. You can run a job once or choose how frequently you would like the job to run. If you will run the job multiple times, you can save the job in Enterprise Manager's jobs library so that it can be rerun in the future. In Figure 5.10, the job will be scheduled to run once, at 11:31 p.m.

Click To expand
Figure 5.10: Scheduling the load.
Monitoring progress of the load operation

You can monitor the progress of a job while it is running by clicking on jobs in the navigator tree on the Enterprise Manager Console, as shown in Figure 5.11. The Active page lists all jobs that are either running or are scheduled to run. By clicking on the job in the Active Jobs page, information about the job's state and progress can be displayed. Load 0016 has been submitted and will run at 11:31 p.m. The History page contains a list of jobs that have been run previously. You can check to see if the job ran successfully after it has completed and look at the output to see any errors when a job has failed.

Click To expand
Figure 5.11: Monitoring jobs from the OEM console.
Inspecting the SQL*Loader log

The following example shows a portion of the log file from a sample load session. A total of 159 rows were appended to the product table. Ten records were rejected due to invalid data. Copies of those records can be found in the file product.bad.

SQL*Loader: Release 9.2.0.1.0 -
Production on Mon Jun 24 06:45:47 2002

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

Control File:   C:\hillson\product.ctl
Data File:      c:\hillson\product.dat
Bad File:     C:\hillson\product.BAD
Discard File:  none specified

 (Allow all discards)

Number to load: ALL
Number to skip: 0
Errors allowed: 50
Bind array:     64 rows, maximum of 256000 bytes
Continuation:    none specified
Path used:      Conventional

Table EASYDW.PRODUCT, loaded from every logical record.
Insert option in effect for this table: APPEND

    Column Name  Position   Len  Term  Encl Datatype
---------------- ---------- ----- ---- ---- ----------
PRODUCT_ID            FIRST     *   ,  O(') CHARACTER
PRODUCT_NAME           NEXT     *   ,  O(') CHARACTER
CATEGORY               NEXT     *   ,  O(') CHARACTER
COST_PRICE             NEXT     *   ,  O(') CHARACTER
SELL_PRICE             NEXT     *   ,  O(') CHARACTER
WEIGHT                 NEXT     *   ,  O(') CHARACTER
SHIPPING_CHARGE        NEXT     *   ,  O(') CHARACTER
MANUFACTURER           NEXT     *   ,  O(') CHARACTER
SUPPLIER               NEXT     *   ,  O(') CHARACTER

Record 1: Rejected - Error on table EASYDW.PRODUCT,
column PRODUCT_ID. Column not found before end of logical record
(use TRAILING NULLCOLS)
Record 3: Rejected - Error on table EASYDW.PRODUCT,
column PRODUCT_ID. Column not found before end of logical record
(use TRAILING NULLCOLS)
Record 4: Rejected - Error on table EASYDW.PRODUCT,
column WEIGHT. Column not found before end of logical record
(use TRAILING NULLCOLS)
Record 5: Rejected - Error on table EASYDW.PRODUCT,
column SELL_PRICE. Column not found before end of logical record
(use TRAILING NULLCOLS)
Record 6: Rejected - Error on table EASYDW.PRODUCT, column
SHIPPING_CHARGE. Column not found before end of logical record
(use TRAILING NULLCOLS)
Record 7: Rejected - Error on table EASYDW.PRODUCT,
column COST_PRICE. Column not found before end of logical record
(use TRAILING NULLCOLS)
Record 8: Rejected - Error on table EASYDW.PRODUCT, column
SHIPPING_CHARGE. Column not found before end of logical record
(use TRAILING NULLCOLS)
Record 9: Rejected - Error on table EASYDW.PRODUCT,
column COST_PRICE. Column not found before end of logical record
(use TRAILING NULLCOLS)
Record 106: Rejected - Error on table EASYDW.PRODUCT.
ORA-00001: unique constraint (EASYDW.PK_PRODUCT) violated

Record 134: Rejected - Error on table EASYDW.PRODUCT.
ORA-00001: unique constraint (EASYDW.PK_PRODUCT) violated

Table EASYDW.PRODUCT:
  159 Rows successfully loaded.
  10 Rows not loaded due to data errors.
  0 Rows not loaded because all WHEN clauses were failed.
  0 Rows not loaded because all fields were null.

Space allocated for bind array:  148608 bytes(64 rows)
Read   buffer bytes: 1048576

Total logical records skipped:          0
Total logical records read:           169
Total logical records rejected:        10
Total logical records discarded:        0

Run began on Mon Jun 24 06:45:47 2002
Run ended on Mon Jun 24 06:46:14 2002

Elapsed time was:     00:00:27.56
CPU time was:         00:00:00.09

Optimizing SQL*Loader performance

When loading large amounts of data in a small batch window, a variety of techniques can be used to optimize performance:

  • Using direct path load. Formatting the data blocks directly and writing them to the database files eliminates much of the work needed to execute a SQL insert statement. Direct path load requires exclusive access to the table or partition being loaded. In addition, triggers are automatically disabled, and constraint evaluation is deferred until the load completes.

  • Disabling integrity constraint evaluation prior to loading the data. When loading data with direct path, SQL*Loader automatically disables all CHECK and REFERENCES integrity constraints. When using parallel direct path load or loading into a single partition, other types of constraints must be disabled. You can manually disable evaluation of not null, unique, and primary-key constraints during the load process, as well. When the load completes, you can have SQL*Loader reenable the constraints, or do it yourself manually.

  • Loading the data in sorted order. Presorting data minimizes the amount of temporary storage needed during the load, enabling optimizations to minimize the processing during the merge phase to be applied. To tell SQL*Loader what indexes the data is sorted on, use the SORTED INDEXES statement in the control file.

  • Deferring index maintenance. Indexes are maintained automatically whenever data is inserted or deleted or the key column is updated. When loading large amounts of data with direct path load, it may be faster to defer index maintenance until after the data is loaded. You can either drop the indexes prior to the beginning of the load or skip index maintenance by setting SKIP_INDEX_MAINTENANCE= TRUE on the SQL*Loader command line. Index partitions that would have been updated are marked "index unusable," because the index segment is inconsistent with respect to the data it indexes. After the data is loaded, the indexes must be rebuilt.

  • Disabling redo logging by using the UNRECOVERABLE option in the control file. By default, all changes made to the database are also written to the redo log so they can be used to recover the database after failures. Media recovery is the process of recovering after the loss of a database file, often due to a hardware failure such as a disk head crash. By disabling redo logging, the load is faster.

    However, if the system fails in the middle of loading the data, you need to restart the load, since you cannot use the redo log for recovery. If you are using Oracle Data Guard to protect your data with a logical or physical standby database, you may not want to disable redo logging. Any data not logged cannot be automatically applied to the standby site.

    After the data is loaded, using the UNRECOVERABLE option, it is important to do a backup to make sure you can recover the data in the future if the need arises.

  • Loading the data into a single partition. While you are loading a partition of a partitioned or subpartitioned table, other users can continue to access the other partitions in the table. Loading the April transactions will not prevent users from querying the existing data for January through March. Thus, overall availability of the warehouse is increased.

  • Loading the data in parallel. When a table is partitioned, it can be loaded into multiple partitions in parallel. You can also set up multiple, concurrent sessions to perform a load into the same table or into the same partition of a partitioned table.

  • Increasing the STREAMSIZE parameter can lead to better direct path load times, since larger amounts of data will be passed in the data stream from the SQL*Loader client to the Oracle server.

  • If the data being loaded contains many duplicate dates, using the DATE_CACHE parameter can lead to better performance of direct path load. Use the date cache statistics (entries, hits, and misses) contained in the SQL*Loader log file to tune the size of the cache for future similar loads.

SQL*Loader direct path load of a single partition

Next, we will look at an example of loading data into a single partition. In the EASYDW warehouse, the fact table is partitioned by date. At the end of April, the April sales transactions are loaded into the EASYDW warehouse.

In the following example, we create a tablespace, add a partition to the purchases table, and then use SQL*Loader direct path load to insert the data into the April 2002 partition.

Step 1: Create a tablespace
SQL> CREATE TABLESPACE purchases_apr2002
     DATAFILE 'c:\ora9ir2\oradata\orcl\PURCHASESAPR2002.f' SIZE 5M
     REUSE AUTOEXTEND ON
     DEFAULT STORAGE
     (INITIAL 64K NEXT 64K PCTINCREASE 0 MAXEXTENTS UNLIMITED);
Step 2: Add a partition
SQL> ALTER TABLE easydw.purchases
     ADD PARTITION purchases_apr2002
     VALUES LESS THAN (TO_DATE('30-04-2002', 'DD-MM-YYYY'))
     PCTFREE 0 PCTUSED 99
     STORAGE (INITIAL 64K NEXT 64K PCTINCREASE 0)
     TABLESPACE purchases_apr2002;
Step 3: Disable all referential integrity constraints and triggers

When using direct path load of a single partition, referential and check constraints on the table partition must be disabled, along with any triggers.

SQL> ALTER TABLE purchases DISABLE CONSTRAINT fk_time;
SQL> ALTER TABLE purchases DISABLE CONSTRAINT fk_product_id;
SQL> ALTER TABLE purchases DISABLE CONSTRAINT fk_customer_id;

The status column in the user_constraints view can be used to determine if the constraint is currently enabled or disabled. Here we can see that the SPECIAL_OFFER constraint is still enabled.

SQL> SELECT TABLE_NAME, CONSTRAINT_NAME, STATUS
     FROM USER_CONSTRAINTS
     WHERE TABLE_NAME = 'PURCHASES';

TABLE_NAME  CONSTRAINT_NAME       STATUS
----------- --------------------- --------
PURCHASES   NOT_NULL_PRODUCT_ID   DISABLED
PURCHASES   NOT_NULL_TIME         DISABLED
PURCHASES   NOT_NULL_CUSTOMER_ID  DISABLED
PURCHASES   SPECIAL_OFFER         ENABLED
PURCHASES   FK_PRODUCT_ID         DISABLED
PURCHASES   FK_TIME               DISABLED
PURCHASES   FK_CUSTOMER_ID        DISABLED

7 rows selected.opt

The status column in the user_triggers view can be used to determine if any triggers must be disabled. There are no triggers on the PURCHASES table.

SQL> SELECT TRIGGER_NAME, STATUS
     FROM ALL_TRIGGERS WHERE TABLE_NAME = 'PURCHASES';

no rows selected
Step 4: Load the data

The following example shows the SQL*Loader control file to load new data into a single partition. Note that the partition clause is used.

UNRECOVERABLE LOAD DATA
INFILE 'purchases.dat' BADFILE 'purchases.bad'
APPEND
INTO TABLE purchases
PARTITION (purchases_apr2002)
 (product_id         position (1-6)        char,
 time_key            position (7-17)       date   "DD-MON-YYYY",
 customer_id         position (18-25)      char,
 purchase_date       position (26-36)      date "DD-MON-YYYY",
 purchase_time       position (37-40)      integer external,
 purchase_price      position (41-45)      decimal external,
 shipping_charge     position (46-49)      integer external,
 today_special_offer position (50)         char)

The unrecoverable keyword is specified, disabling media recovery for the table being loaded. Database changes being made by other users will continue to be logged. After disabling media recovery, it is important to do a backup, to make it possible to recover the data in the future if the need arises. If you attempted media recovery before the backup was taken, you would discover that the data blocks that were loaded have been marked as logically corrupt. To recover the data, you must drop the partition, and reload the data.

Any data that cannot be loaded will be written to the file purchases.bad. Data will be loaded into the purchases_apr2002 partition. This example shows loading a fixed-length file named purchases.dat. Each field in the input record is described by specifying its starting position, its ending position, and its data type. Note that these are SQL*Loader data types, not the data types in an Oracle table. When the data is loaded into the tables, each field is converted to the data type of the column, if necessary.

The following example shows a sample of the purchases data file. The PRODUCT_ID starts in column 1 and is six bytes long. Time_key starts in column 7 and is 17 bytes long. The data mask DD-MON-YYYY is used to describe the input format of the date fields.

123456789012345678901234567890123456789012345678901
      |          |       |          |    |   |   |
SP100101-APR-2002AB12345601-APR-2002002428.014.50Y
SP100101-APR-2002AB12345701-APR-2002102428.014.50Y

To invoke SQL*Loader direct path mode from the command line, set direct=true. In this example, skip_index_maintenance is set to true, so the indexes will need to be rebuilt after the load.

sqlldr USERID=easydw/easydw CONTROL=purchases.ctl
LOG=purchases.log DIRECT=TRUE SKIP_INDEX_MAINTENANCE = TRUE
Step 5: Inspect the log

The following example shows a portion of the SQL*Loader log file from the load operation. Rather than generating redo to allow recovery, invalidation redo was generated to let Oracle know this table cannot be recovered. The indexes were made unusable. The column starting position and length are described.

SQL*Loader: Release 9.2.0.1.0 -
Production on Sun Jun 9 12:43:45 2002

Copyright (c) 1982, 2002, Oracle Corporation.
All rights reserved.

Control File:   purchases.ctl
Data File:      purchases.dat
  Bad File:     purchases.bad
  Discard File:  none specified

 (Allow all discards)

Number to load: ALL
Number to skip: 0
Errors allowed: 50
Continuation:    none specified
Path used:      Direct

Load is UNRECOVERABLE; invalidation redo is produced.

Table PURCHASES, partition PURCHASES_APR2002,
loaded from every logical record.
Insert option in effect for this partition: APPEND

   Column Name  Position   Len  Term Encl Datatype
--------------- ---------- ---- ---- ---- --------
PRODUCT_ID          1:6     6            CHARACTER
TIME_KEY            7:17    11           DATE DD-MON-YYYY
CUSTOMER_ID        18:25     8           CHARACTER
PURCHASE_DATE      26:36    11           DATE DD-MON-YYYY
PURCHASE_TIME      37:40     4           CHARACTER
PURCHASE_PRICE     41:45     5           CHARACTER
SHIPPING_CHARGE    46:49     4           CHARACTER
TODAY_SPECIAL_OFFER   50     1           CHARACTER
The following index(es) on table PURCHASES were processed:
index EASYDW.PURCHASE_PRODUCT_INDEX partition PURCHASES_APR2002
was made unusable due to:
SKIP_INDEX_MAINTENANCE option requested
index EASYDW.PURCHASE_CUSTOMER_INDEX partition PURCHASES_APR2002
was made unusable due to:
SKIP_INDEX_MAINTENANCE option requested
index EASYDW.PURCHASE_SPECIAL_INDEX partition PURCHASES_APR2002
was made unusable due to:
SKIP_INDEX_MAINTENANCE option requested
Step 6: Reenable all constraints and triggers and rebuild indexes

After loading the data into a single partition, all references to constraints and triggers must be reenabled. All local indexes for the partition can be maintained by SQL*Loader. Global indexes are not maintained on single partition or subpartition direct path loads and must be rebuilt. In the previous example, the indexes must be rebuilt since index maintenance was skipped. These steps are discussed in more detail later in the chapter.

SQL*Loader parallel direct path load

When a table is partitioned, the direct path loader can be used to load multiple partitions in parallel. Each parallel direct path load process should be loaded into a partition of a table stored on a separate disk, to minimize I/O contention.

Since data is extracted from multiple operational systems, you will often have several input files that need to be loaded into the warehouse. These files can be loaded in parallel and the workload distributed among several concurrent SQL*Loader sessions.

Figure 5.12 shows an example of how parallel direct path load can be used to initially load the historical transactions into the purchases table. You need to invoke multiple SQL*Loader sessions. Each SQL*Loader session takes a different data file as input. In this example, there are three data files, each containing the purchases for one month: January, February, and March. These will be loaded into the purchases table, which is also partitioned by month. Each data file is loaded in parallel into its own partition.

Click To expand
Figure 5.12: SQL*Loader parallel direct path load.

It is suggested that the following steps are followed to load data in parallel using SQL*Loader:

Step 1: Disable all constraints and triggers

Constraints cannot be evaluated, and triggers cannot be fired during a parallel direct path load. If you forget, SQL*Loader will issue an error.

Step 2: Drop all indexes

Indexes cannot be maintained during a parallel direct path load. However, if we are loading only a few partitions out of many, then it is probably better to skip index maintenance and have them marked as unusable instead.

Step 3: Load the data

By invoking multiple SQL*Loader sessions and setting direct and parallel to true, the processes will load concurrently. Depending on your operating system, you may need to put an '&' at the end of each line.

sqlldr userid=easydw/easydw
CONTROL=jan.ctl DIRECT=TRUE PARALLEL=TRUE
sqlldr userid=easydw/easydw
CONTROL=feb.ctl DIRECT=TRUE PARALLEL=TRUE
sqlldr userid=easydw/easydw
CONTROL=mar.ctl DIRECT=TRUE PARALLEL=TRUE
Step 4: Inspect the log

A portion of one of the log files follows. Note that the mode is direct with the parallel option.

SQL*Loader: Release 9.2.0.1.0 - Production on
Sat Jun 15 11:01:21 2002
Copyright (c) 1982, 2002, Oracle Corporation.
All rights reserved.

Control File:   c:\feb.ctl
Data File:      c:\feb.dat
  Bad File:     c:\feb.bad
  Discard File:  none specified

 (Allow all discards)

Number to load: ALL
Number to skip: 0
Errors allowed: 50
Continuation:    none specified
Path used:      Direct - with parallel option.

Load is UNRECOVERABLE; invalidation redo is produced.

Table PURCHASES, partition PURCHASES_FEB2002,
loaded from every logical record.
Insert option in effect for this partition: APPEND
Step 5: Reenable all constraints and triggers

Recreate all indexes. After using parallel direct path load, reenable any constraints and triggers that were disabled for the load. Recreate any indexes that were dropped.

Transformations using SQL*Loader

If you receive extract files that have data in them that you do not want toload into the warehouse, you can use SQL*Loader to filter the rows of interest. You select the records that meet the load criteria by specifying a WHENclause to test an equality or inequality condition in the record. If a record does not satisfy the WHEN condition, it is written to the discard file. The discard file contains records that were filtered out of the load because they did not match any record-selection criteria specified in the control file. Note that these records differ from rejected records written to the BAD file. Discarded records do not necessarily have any bad data. The WHEN clause can be used with either conventional or direct path load.

You can use SQL*Loader to perform simple types of transformations on character data. For example, portions of a string can be inserted using the substring function; two fields can be concatenated together using the CONCAT operator. You can trim leading or trailing characters from a string using the trim operator. The control file in the following example illustrates the use of the WHEN clause to discard any rows where the PRODUCT_ID is blank; it also shows how to uppercase the PRODUCT_ID column.

LOAD DATA
INFILE 'product.dat' append
INTO TABLE product WHEN product_id != BLANKS
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY "'"
(product_id "upper(:product_id)",
 product_name,
 category,
 cost_price,
 sell_price,
 weight,
 shipping_charge,
 manufacturer,
 supplier)

The discard file is specified when invoking SQL*Loader from the command line, as shown, or with the OEM Load Wizard, as seen previously.

sqlldr userid=easydw/easydw CONTROL=product.ctl
LOG=product.log BAD=product.bad DISCARD=product.dis DIRECT=true

In Oracle 9i these types of transformations can be done with both direct path and conventional path modes. However, since they are applied to each record individually, they do have an impact on the load performance.

SQL*Loader postload operations

After the data is loaded, you may need to process exceptions, reenable constraints, and rebuild indexes.

Inspect the logs

Always look at the logs to ensure the data was loaded successfully. Validate that the correct number of rows have been added.

Process the load exceptions

Look in the .bad file to find out which rows were not loaded. Records that fail NOT NULL constraints are rejected and written to the SQL*Loader bad file.

Reenable data integrity constraints

Ensure referential integrity if you have not already done so. Make sure that each foreign key in the fact table has a corresponding primary key in each dimension table. For the EASYDW warehouse, each row in the PURCHASES table needs to have a valid CUSTOMER_ID in the CUSTOMERS table and a valid PRODUCT_ID in the PRODUCTS table.

When using direct path load, CHECK and REFERENCES integrity constraints were disabled. When using parallel direct path load, all constraints were disabled.

Handle constraint violations

To find the rows with bad data, you can create an exceptions table. Create the table named "exceptions" by running the script UTLEXCPT.SQL. When enabling the constraint, list the table name the exceptions should be written to.

SQL> ALTER TABLE purchases ENABLE CONSTRAINT fk_product_id
                           EXCEPTIONS INTO exceptions;

In our example, two rows had bad PRODUCT_IDs. A sale was made for a product that does not exist in the product dimension.

SQL> SELECT * FROM EXCEPTIONS;

ROW_ID                  OWNER      TABLE_NAME  CONSTRAINT
------------------------------------------------------------
AAAC/ZAAMAAAAADAAF      EASYDW     PURCHASES   FK_PRODUCT_ID
AAAC/ZAAMAAAAADAAI      EASYDW     PURCHASES   FK_PRODUCT_ID

To find out which rows have violated referential integrity, select the rows from the purchases table where the rowid is in the exception table. In this example, there are two rows where there is no matching product in the products dimension.

SQL> SELECT * from purchases WHERE row_id in (select row_id from
exceptions);

PRODUCT_ TIME_KEY CUSTOMER_I PURCHASE_ PURCHASE_TIM PURCHASE_PRICE
------- --------- ---------- --------- ------------ -------------
XY1001  01-MAR-02 AB123457  01-MAR-02   1024        28.01 4.5 Y
AB1234  01-MAR-02 AB123456  01-MAR-02     24        28.01 4.5 Y
Hint:

It is important to fix any referential integrity constraint problems, particularly if you are using Summary Management, which relies on the correctness of these relationships to perform query rewrite.
Enabling constraints without validation

If integrity checking is maintained in an application or the data has already been cleansed, and you know it will not violate any integrity constraints, enable the constraints with the NOVALIDATE clause. Include the rely clause for query rewrite. Since summary management and other tools depend on the relationships defined by references constraints, you should always define the constraints, even if they are not validated.

SQL>ALTER TABLE purchases ENABLE NOVALIDATE CONSTRAINT fk_product_id;
SQL>ALTER TABLE purchases MODIFY CONSTRAINT fk_product_id RELY;
Check for unusable indexes

Prior to publishing data in the warehouse, you should check to determine if any indexes are in an unusable state. An index becomes unusable when it no longer contains index entries to all the data. An index may be marked unusable for a variety of reasons, including the following:

  • You requested that index maintenance be deferred, using the skip_index_maintenance parameter when invoking SQL*Loader.

  • UNIQUE constraints were not disabled when using SQL*Loader direct path. At the end of the load, the constraints are verified when the indexes are rebuilt. If any duplicates are found, the index is not correct and will be left in an "Index Unusable" state.

The index must be dropped and recreated or rebuilt to make it usable again. If one partition is marked UNUSABLE, the other partitions of the index are still valid.

To check for unusable indexes query the table USER_INDEXES as illustrated in the following code. Here we can see that the PRODUCT_PK_INDEX is unusable. If an index is partitioned, its status is N/A.

SQL> SELECT INDEX_NAME, STATUS FROM USER_INDEXES;

INDEX_NAME              STATUS
-------------------------------
CUSTOMER_PK_INDEX       VALID
I_SNAP$_CUSTOMER_SUM    VALID
PRODUCT_PK_INDEX        UNUSABLE
PURCHASE_CUSTOMER_INDEX N/A
PURCHASE_PRODUCT_INDEX  N/A
PURCHASE_SPECIAL_INDEX  N/A
PURCHASE_TIME_INDEX     N/A
TIME_PK_INDEX           VALID
TSO_PK_INDEX            VALID

9 rows selected.

Next check to see if any index partitions are in an unusable state by checking the table USER_IND_PARTITIONS. In the following example the PURCHASE_PRODUCT_INDEX for the "purchases_apr2002" partition is unusable.

SQL> SELECT INDEX_NAME, PARTITION_NAME, STATUS FROM
     USER_IND_PARTITIONS WHERE STATUS != 'VALID';

INDEX_NAME               PARTITION_NAME      STATUS
------------------------ ------------------- --------
PURCHASE_PRODUCT_INDEX   PURCHASES_JAN2002   USABLE
PURCHASE_PRODUCT_INDEX   PURCHASES_FEB2002   USABLE
PURCHASE_PRODUCT_INDEX   PURCHASES_MAR2002   USABLE
PURCHASE_PRODUCT_INDEX   PURCHASES_APR2002   UNUSABLE
Rebuild unusable indexes

If you have any indexes that are unusable, you must rebuild them prior to accessing the table or partition. In the following example, the purchases_product_index for the newly added purchases_apr2002 partition is rebuilt.

SQL> ALTER INDEX purchase_product_index
     REBUILD PARTITION purchases_apr2002;

5.4.2 Loading the warehouse using external tables

An external table is a table stored outside the database in a flat file. The data in an external table can be queried just like a table stored inside the database. You can select columns, rows, and join the data to other tables using SQL. The data in the external table can be accessed in parallel, just like tables stored in the database.

External tables are read-only. No DML operations are allowed, and you cannot create indexes on an external table. If you need to update the data or access it more than once, you can load the data into the database, where you can update it and add indexes to improve query performance. By loading the data into the database, you can manage the data as part of the database. RMAN will not back up the data for any external tables.

Since you can query the data in the external table you have just defined using SQL, you can also load the data using an INSERT SELECT statement. External tables provide an alternative to SQL*Loader to load data from flat files. They can be used to perform more complex transformations while the data is loaded and simplify many of the operational aspects while loading data in parallel and managing triggers, constraints, and indexes. However, in Oracle 9i SQL*Loader direct path load may still be faster in many cases. There are many similarities between the two methods, and if you've used SQL*Loader, you will find it easy to learn to use external tables.

Specifying the location of the data file and log files for an external table

Before creating an external table, a directory object must be created to specify the location of the data files and log files. The data and log files must be on the same machine as the database server, or must be accessible to the server. This is different from SQL*Loader, where the SQL*Loader client sends the data to the server. Directories are created in a single namespace and are not owned by an individual's schema. In the following example, the DBA creates directories for the data and log files.

SQL> CREATE OR REPLACE DIRECTORY data_file_dir as 'C:\datafiles\';
SQL> CREATE OR REPLACE DIRECTORY log_file_dir as 'C:logfiles\';

Read access must be granted to users who will be accessing the data files in the directory. In the following example, the DBA grants read access to user EASYDW.

SQL> GRANT READ ON DIRECTORY data_file_dir to easydw;
SQL> GRANT READ ON DIRECTORY log_file_dir to easydw;

Looking at the DBA_DIRECTORIES view, we can see the directories that have been defined.

SQL> SELECT * FROM DBA_DIRECTORIES;

OWNER      DIRECTORY_NAME   DIRECTORY_PATH
---------- ---------------- --------------

SYS        LOG_FILE_DIR     c:\logfiles\

SYS        DATA_FILE_DIR    c:\datafiles\

Creating an external table

In order to create an external table you must specify the following:

  • The metadata, which describes how the data looks to Oracle, including the external table name, the column names, and Oracle data types. These are the names you will use in the SQL statements to access the external table. Metadata is stored in the Oracle data dictionary. In the example shown in Figure 5.13, the external table name is NEW_PRODUCTS. The data is stored outside the database. This differs from the other regular tables in the database, where the data and metadata are both stored in the database.

    Click To expand
    Figure 5.13: External table.

  • The access parameters, which describe how the data is stored in the external file, where it is located, its format, and how to identify the fields and records. The information is similar to what is stored in a SQL*Loader control file. The access driver uses the access parameters. An example of the access parameters is shown in the CREATE EXTERNAL TABLE statement.

In Oracle 9i there is only one type of access driver, called ORACLE_LOADER, which provides read-only access to flat files. It is specified in the TYPE clause on the CREATE EXTERNAL TABLE statement.

The following example shows the SQL to create an external table. The CREATE TABLE ORGANIZATION EXTERNAL clause is used. The metadata describing the column names and data types are listed. The TYPE clause lists the type of access driver and begins the description of how the file is stored on disk. Each field is separated by a comma and optionally enclosed in an apostrophe. The location of the data file, log file, and bad files is specified. By specifying the parallel clause, the data will be loaded in parallel.

CREATE TABLE new_products
(product_id             VARCHAR2(8),
 product_name           VARCHAR2(30),
 category               VARCHAR2(4),
 cost_price             NUMBER (6,2),
 sell_price             NUMBER (6,2),
 weight                 NUMBER (4,2),
 shipping_charge        NUMBER (5,2),
 manufacturer           VARCHAR2(20),
 supplier               VARCHAR2(10))
ORGANIZATION EXTERNAL
(TYPE ORACLE_LOADER
   DEFAULT DIRECTORY data_file_dir
   ACCESS PARAMETERS (RECORDS DELIMITED BY NEWLINE
                    CHARACTERSET US7ASCII
                    BADFILE log_file_dir:'product.bad'
                    LOGFILE log_file_dir:'product.log'
                    FIELDS TERMINATED BY ','
                    OPTIONALLY ENCLOSED BY "'")
                    LOCATION ('product.dat'))
                    REJECT LIMIT UNLIMITED PARALLEL;

A portion of the data file, product.dat is shown in the following example. Each field is separated by a comma and optionally enclosed in a single quote. As the data is read, each field in the input file is mapped to the corresponding columns in the external table definition. As the data is read, it is converted from the data type of the input file to the data type of the column in the database as necessary.

'SP1000','Camera','ELEC',45.67,67.23,15.00,4.50,'Ricoh','Ricoh'
'SP1001','APS Camera','ELEC',24.67,36.23,5.00,4.50,'Ricoh','Ricoh'
'SP1010','Camera','ELEC',35.67,47.89,5.00,4.50,'Agfa','Agfa'

After executing the CREATE TABLE command shown previously, the metadata for the NEW_PRODUCTS table is stored in the database. The table is as follows.

SQL> DESCRIBE new_products;

 Name                  Null?    Type
 --------------------- -------- ------------
 PRODUCT_ID                     VARCHAR2(8)
 PRODUCT_NAME                   VARCHAR2(30)
 CATEGORY                       VARCHAR2(4)
 COST_PRICE                     NUMBER(6,2)
 SELL_PRICE                     NUMBER(6,2)
 WEIGHT                         NUMBER(4,2)
 SHIPPING_CHARGE                NUMBER(5,2)
 MANUFACTURER                   VARCHAR2(20)
 SUPPLIER                       VARCHAR2(10)

The USER_EXTERNAL_TABLES dictionary view shows what external tables have been created, along with a description of them. The USER_EXTERNAL_LOCATIONS dictionary view shows the location of the data file.

SQL> SELECT TABLE_NAME, TYPE_NAME, DEFAULT_DIRECTORY_NAME
     FROM USER_EXTERNAL_TABLES;

T0ABLE_NAME   TYPE_NAME       DEFAULT_DIRECTORY_NAME
------------- --------------- ----------------------
NEW_PRODUCTS  ORACLE_LOADER    DATA_FILE_DIR

SQL> SELECT * FROM USER_EXTERNAL_LOCATIONS;

TABLE_NAME    LOCATION     DIR DIRECTORY_NAME
------------- ------------ --- --------------
NEW_PRODUCTS  product.dat  SYS DATA_FILE_DIR

Accessing data stored in an external table

After defining the external table, it can be accessed using SQL just as if it were any other table in the database, although the data is being read from the file outside the database. A portion of the output is as follows:

SQL> SELECT * FROM NEW_PRODUCTS;

PRODUCT PRODUCT       CAT COST  SELL WEIGHT SHIPPING MANUF SUPPL
ID      NAME              PRICE PRICE       CHARGE
------------------------------------------------------------------
SP1000 Camera         ELEC 45.67 67.23 15     4.5    Ricoh Ricoh
SP1001 APS  Camera    ELEC 24.67 36.23  5     4.5    Ricoh Ricoh
SP1010 Camera         ELEC 35.67 47.89  5     4.5    Agfa Agfa

Loading data from an external table

The next example will use the INSERT/SELECT statement to load the data into the PRODUCT dimension table. It is very easy to perform transformations during the load using SQL functions and arithmetic operators. In this example, after the data file was created, the cost of fuel rose, and our shipping company increased the shipping rates by 10 percent. In this example, data is loaded into the EASYDW.PRODUCT table by selecting the columns from the NEW_PRODUCTS external table. The shipping charge is increased by 10 percent for each item as the data is loaded.

SQL> INSERT INTO easydw.product
           (product_id, product_name, category, cost_price,
            sell_price, weight, shipping_charge, manufacturer,
            supplier)
     SELECT product_id, product_name, category, cost_price,
            sell_price, weight,
            (shipping_charge + (shipping_charge * .10)),
            manufacturer, supplier FROM new_products;

The data has now been loaded into the EASYDW.PRODUCT table, and we can see that the shipping charge increased from $4.50 to $4.95 for the items displayed.

SQL> select * from easydw.product;

PRODUCT PRODUCT       CAT COST  SELL WEIGHT SHIPPING MANUF SUPPL
ID      NAME              PRICE PRICE       CHARGE
------------------------------------------------------------------
SP1000 Camera         ELEC 45.67 67.23 15     4.95   Ricoh Ricoh
SP1001 APS Camera     ELEC 24.67 36.23  5     4.95   Ricoh Ricoh
SP1010 Camera         ELEC 35.67 47.89  5     4.95   Agfa Agfa

Loading data in parallel using external tables

Another advantage of using external tables is the ability to load the data in parallel without having to split a large file into smaller files and start multiple sessions, as you must do with SQL*Loader. The degree of parallelism is set using the standard parallel hints or with the PARALLEL clause when creating the external table, as shown previously. The output of an EXPLAIN PLAN shows the parallel access.

SQL> EXPLAIN PLAN FOR INSERT INTO easydw.product
           (product_id, product_name, category, cost_price,
            sell_price, weight, shipping_charge, manufacturer,
            supplier)
     SELECT product_id, product_name, category, cost_price,
            sell_price, weight,
            (shipping_charge + (shipping_charge * .10)),
            manufacturer, supplier FROM new_products;

The utlxplp.sql procedure is used to show the EXPLAIN PLAN output with the columns pertaining to parallel execution, which have been edited and and are shown in bold type for readability.

SQL> @c:\ora9iR2\rdbms\admin\utlxplp.sql

PLAN_TABLE_OUTPUT
----------------------------------------------------------------
|Id|Operation|Name    |Rows|Bytes|Cost |TQ   |IN-OUT|PQ Distrib|
----------------------------------------------------------------
| 0|INSERT   |        |8168| 781K|   4 |     |      |          |
| 1|EXTERNAL |NEW_    |8168| 781K|   4 |20,00|P->S  |QC        |
|  |TABLE    |PRODUCTS|    |     |     |     |      |RANDOM    |
|  |ACCESS   |        |    |     |     |     |      |          |
|  |FULL     |        |    |     |     |     |      |          |
----------------------------------------------------------------

Loading the dimensions using SQL MERGE

When loading new data into existing dimension tables, you may need to add new rows and make changes to existing rows. In the past, special programming logic was required to differentiate a new row from a changed row. A new capability was added in Oracle 9i, which makes this process much easier: the SQL MERGE. This is often called an "upsert." If a row exists, update it. If it doesn't exist, insert it.

In the following example, new data is added to the customer dimension. The input file has both new customers who have been added, as well as changes to existing customer data. In this case, there is no need to retain the old customer information, so it will be updated. An external table, named CUSTOMER_CHANGES, is created for the file containing the updates to the customer dimension.

CREATE TABLE easydw.customer_changes
(customer_id            VARCHAR2(10),
 town                   VARCHAR2(10),
 county                 VARCHAR2(10),
 postal_code            VARCHAR2(10),
 dob                    DATE,
 country                VARCHAR2(20),
 occupation             VARCHAR2(10))
ORGANIZATION EXTERNAL
(TYPE oracle_loader
      DEFAULT DIRECTORY data_file_dir
      ACCESS PARAMETERS (RECORDS DELIMITED BY NEWLINE
                         CHARACTERSET US7ASCII
                         BADFILE log_file_dir:'cus_changes.bad'
                         LOGFILE log_file_dir:'cust_changes.log'
                         FIELDS TERMINATED BY ','
                         OPTIONALLY ENCLOSED BY "'")
                         LOCATION ('customer_changes.dat'))
                         REJECT LIMIT UNLIMITED NOPARALLEL;

Instead of using an insert statement to load the data, the MERGE statement is used. The MERGE statement has two parts. When the customer_id in the customer_table matches the customer_id in the customer_changes table, the row is updated. When the customer_id's do not match, a new row is inserted.

MERGE INTO easydw.customer c
USING easydw.customer_changes cc
ON (c.customer_id = cc.customer_id)
WHEN MATCHED THEN UPDATE SET
c.town=cc.town,
c.county=cc.county,
c.postal_code=cc.postal_code,
      c.dob=cc.dob,
c.country=cc.country,
c.occupation=cc.occupation
WHEN NOT MATCHED THEN INSERT
     (customer_id, town, county, postal_code,
      dob, country, occupation)
     VALUES
     (cc.customer_id, cc.town, cc.county, cc.postal_code,
      cc.dob, cc.country, cc.occupation);

Before merging the customer changes, we had 45 customers.

SQL> SELECT COUNT(*) FROM customer;

  COUNT(*)
----------
        45

Once the external table is created, we can use SQL to look at the customer_changes data file. The first two are rows that will be updated. The first customer was previously a housewife and is now returning to work as an engineer. The second customer moved from Soton2 to Soton. The last two rows are new customers who will be inserted.

SQL> SELECT * FROM customer_changes;

CUSTOMER_ID TOWN   COUNTY  POSTAL_CODE DOB    COUNTRY  OCCUPATION
----------- ------ ------- ----------- ------ -------- ----------
AB123459   Soton   Hants   SO11TF     23-SEP-27 UK      ENGINEER
AB123460   Soton   Hants   SO11TF     23-SEP-27 UK      HOUSEWIFE
AA114778   London  London  W11QC      14-APR-56 UK      ENGINEER
AA123478   London  London  W11QC      14-APR-56 UK      ENGINEER

Looking at a portion of the customer dimension, we can see the rows where the CUSTOMER_ID column matches the CUSTOMER_ID column in the CUSTOMER_CHANGES external table. These rows will be updated.

SQL> SELECT * FROM customer;

CUSTOMER_ID TOWN   COUNTY  POSTAL_CODE DOB    COUNTRY  OCCUPATION
----------- ------ ------- ----------- ------ -------- ----------
AB123459   Soton   Hants   SO11TF     23-SEP-27 UK      HOUSEWIFE
AB123460   Soton2  Hants   SO11TF     23-SEP-27 UK      HOUSEWIFE

After the MERGE, we now have 47 customers. A portion of the output, just the rows that have changed, are displayed from the customer table.

SQL> SELECT COUNT(*) FROM customer;

  COUNT(*)
  --------
       47

SQL> SELECT * FROM customer;

CUSTOMER_ID TOWN   COUNTY  POSTAL_CODE DOB    COUNTRY  OCCUPATION
----------- ------ ------- ----------- ------ -------- ----------
AB123459   Soton   Hants   SO11TF     23-SEP-27 UK      ENGINEER
AB123460   Soton   Hants   SO11TF     23-SEP-27 UK      HOUSEWIFE
AA114778   London  London  W11QC      14-APR-56 UK      ENGINEER
AA123478   London  London  W11QC      14-APR-56 UK      ENGINEER

In this example the MERGE statement was used with external tables. It can also be used with any user tables.

5.4.3 Loading the warehouse using transportable tablespaces

The fastest way to move data from one Oracle database to another is by using transportable tablespaces. Transportable tablespaces provide a mechanism to move one or more tablespaces from one Oracle database into another. Rather than processing the data a row at time, the entire file or set of files is physically copied from one database and integrated into the second database by importing the metadata describing the tables in the files. In addition to data in tables, the indexes can also be moved.

Because the data is not unloaded and reloaded, it can be moved quickly, but it cannot be converted to another format. Thus, in Oracle 9i, you can transport tablespaces between Oracle databases that use the same character set and the same operating system. If any types of data conversions are necessary, another mechanism must be used.

Transportable tablespaces can be used to move data from the operational database to the staging area if you are using an Oracle database to do your staging. Transportable tablespaces are also useful to move data from the data warehouse to a dependent data mart.

Figure 5.14 shows the steps involved in using transportable tablespaces. These steps are as follows:

  1. Create a new tablespace.

  2. Move the data you want to transfer into its own tablespace.

  3. Alter the tablespace to read only.

  4. Use the export utility to unload the metadata describing the objects in the tablespace.

  5. Copy the data files and export dump file containing the metadata to the target system.

  6. Use the import utility to load the metadata descriptions into the target database.

  7. Alter the tablespace to read write.

  8. Perform transformations.

  9. Move the data from the staging area to the warehouse fact table.

Click To expand
Figure 5.14: Transportable tablespaces.

This section will talk about steps 1–7. In the next sections, we'll look at steps 8 and 9.

One of the sources for data in the EASYDW warehouse is an Oracle 9i order-processing system. In this example, the order-entry system has a record for every order stored in the purchases table. At the end of April, all the orders for April 2002 will be copied into a new table called APR_ORDERS in the ORDERS tablespace stored in the datafile "orders.f."

Here are the steps that are required to transport a tablespace from one database to another.

Step 1: Create a tablespace in the OLTP system

Choose a name that is unique on both the source and target system. In this example, the tablespace ORDERS is created, and its corresponding data file is "orders.f."

CREATE TABLESPACE orders
datafile 'c:\ora9ir2\oradata\orcl\orders.f' SIZE 5M REUSE
AUTOEXTEND ON
DEFAULT STORAGE
(INITIAL 64K PCTINCREASE 0 MAXEXTENTS UNLIMITED);
Step 2: Move the data for April 2002 into a table in the newly created tablespace

In this example, a table is created and populated using the CREATE TABLE AS SELECT statement. It is created in the ORDERS tablespace. In this example, there are 15,004 orders for April.

SQL> CREATE TABLE apr_orders TABLESPACE orders AS
     SELECT * FROM purchases WHERE
     purchase_date BETWEEN '01-APR-2002' AND '30-APR-2002';

SQL> SELECT COUNT(*) FROM apr_orders;

  COUNT(*)
  --------
    15004

Each tablespace must be self-contained and cannot reference anything outside the tablespace. If there was a global index on the April ORDERS table, it would not be self-contained, and the index would have to be dropped before the tablespace could be moved.

Step 3: Alter the tablespace so that it is read only

If you do not do this, the export in the next step will fail.

ALTER TABLESPACE orders READ ONLY;
Step 4: Export the metadata

Using the EXPORT command, the metadata definitions for the orders tablespace are extracted and stored in the export dump file "expdat.dmp."

exp 'system/manager as sysdba' TRANSPORT_TABLESPACE=y
      TABLESPACES=orders TRIGGERS=n CONSTRAINTS=n GRANTS=n
      FILE=expdat.dmp LOG=export.log

When transporting a set of tablespaces, you can choose to include referential integrity constraints. However, if you do include these, you must move the tables with both the primary and foreign keys.

Hint:

When exporting and importing tablespaces, be sure to connect" as sysdba."

The following example is a copy of the log file, export.log, generated from the export. Only the tablespace metadata is exported, not the data.

Connected to: Oracle9i Enterprise Edition Release 9.2.0.1.0 -
Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.1.0 - Production
Export done in WE8MSWIN1252 character set and AL16UTF16 NCHAR
character set
Note: table data (rows) will not be exported
Note: grants on tables/views/sequences/roles will not be exported
Note: constraints on tables will not be exported
About to export transportable tablespace metadata...
For tablespace ORDERS ...
. exporting cluster definitions
. exporting table definitions
. . exporting table                     APR_ORDERS
. end transportable tablespace metadata export
Export terminated successfully without warnings.
Step 5: Transport the tablespace

Now copy the data file, orders.f, and the export dump file, expdat.dmp, to the physical location on the system containing the staging database. You can use any facility for copying flat files, such as an operating system copy utility or FTP. These should be copied in binary mode, since they are not ASCII files.

In our example, orders.f was copied to c:\ora9ir2\oradata\orcl\orders.f on the staging system using FTP.

Step 6: Import the metadata

By importing the metadata, you are plugging the tablespace into the target database.

imp 'system/manager AS SYSDBA' TRANSPORT_TABLESPACE=Y
Datafiles='c:\ora9ir2\oradata\orcl\orders.f'
FILE=expdat.dmp LOG=import.log

Check the import log to ensure no errors have occurred. Note that the transportable tablespace metadata was imported.

Connected to: Oracle9i Enterprise Edition Release 9.2.0.1.0 -
Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.1.0 - Production

Export file created by EXPORT:V09.02.00 via conventional path
About to import transportable tablespace(s) metadata...
import done in WE8MSWIN1252 character set and AL16UTF16 NCHAR
character set
. importing SYS's objects into SYS
. importing SYSTEM's objects into SYSTEM
. . importing table                   "APR_ORDERS"
Import terminated successfully without warnings.

Check the count to ensure that the totals match the OLTP system.

SQL> SELECT COUNT(*) FROM apr_orders;

  COUNT(*)
----------
     15004
Step 7: Alter the ORDERS tablespace so that it is in read/write mode

You are ready to perform your transformations!

SQL> ALTER TABLESPACE orders READ WRITE;

As you can see by following these steps, the individual rows of a table are never unloaded and reloaded into the database. Thus, using transportable tablespaces is the fastest way to move data between two Oracle databases.


Transformations inside the Oracle Server

5.5 Transformations inside the Oracle Server

Section 5.3 introduced transformations and discussed choosing the optimal place to perform the transformations. We've seen examples of transforming the data while loading it using both SQL*Loader and external tables. This section will discuss performing transformations inside the Oracle Server. If you are doing transformations in the Oracle data warehouse, you typically load data into temporary staging tables, transform the data, and then move this data to the warehouse detail fact tables.

When using transportable tablespaces, as discussed in the previous section, the data was moved from the OLTP system to the staging area in the Oracle warehouse. Data that was loaded using external tables or SQL*Loader can be further transformed in a staging area inside the Oracle Server. Oracle 9i provides tools that can be used to implement transformations in SQL, PL/SQL, or Java stored procedures.

In this section we will first look at the SQL to perform the following simple transformations:

  • Remove the "-" from product ID and increase the shipping charges by 10 percent.

  • Check for invalid product_id's.

  • Look up the warehouse key, and substitute it for the PRODUCT_ID.

Then, we will rewrite the first transformation as a table function.

5.5.1 Transformations that cleanse data and derive new data

The SQL UPDATE statement using built-in functions can be used to perform some simple transformations. Continuing with our example, the APR_ORDERS table must be cleansed. Some of the PRODUCT_ID'S contain a "-", which needs to be removed. The fuel costs have increased, so we must add 10 percent to the shipping charges. Both of these operations can be done in one step using the UPDATE statement. A few of the rows are shown before they are transformed.

SQL> SELECT * FROM apr_orders;

PRODUCT TIME_KEY  CUSTOMER  PURCHASE PURCH PURCH SHIP    TODAYS_
ID                ID        DATE     TIME  PRICE CHARGE  SPECIAL
------- --------- --------- -------- ----- ----- ------ -------
SP1001   02-APR-02 AB123457 02-APR-02 1024 28.01 5.45    N
SP-1000  01-APR-02 AB123456 01-APR-02 1001 67.23 5.45    N
SP-1000  01-APR-02 AB123457 01-APR-02 1002 67.23 5.45    N

The SQL multiplication operator, (*), will be used to update the shipping charge, and the REPLACE function will be used to replace the hyphen, "-", with an empty quote, "", thus removing it. The modified fields are shown in bold type. The shipping charge has increased from $5.45 to $6. The hyphen has been removed from the PRODUCT_ID.

SQL> UPDATE apr_orders SET
shipping_charge = (shipping_charge + shipping_charge *.10),
product_id=REPLACE(product_id, '-','');

PRODUCT TIME_KEY  CUSTOMER  PURCHASE PURCH PURCH SHIP   TODAYS_
ID                ID        DATE     TIME  PRICE CHARGE SPECIAL
------- --------- --------- -------- ----- ----- ------ -------
SP1001  02-APR-02 AB123457 02-APR-02 1024 28.01  6      N
SP1000  01-APR-02 AB123456 01-APR-02 1001 67.23  6      N
SP1000  01-APR-02 AB123457 01-APR-02 1002 67.23  6      N

5.5.2 Validating data using a dimension

Often the incoming data must be validated using information that is in the dimension tables. While this is not actually a transformation, it is discussed here, since it is often done at this stage prior to loading the warehouse tables. In our example, we want to ensure that all the PRODUCT_ID'S for the April orders are valid. The PRODUCT_ID for each order must match a PRODUCT_ID in the product dimension.

This query shows that the April data has an invalid PRODUCT_ID, where there is no matching PRODUCT_CODE in the product dimension. Any data that is invalid should be corrected prior to loading it from the staging area into the warehouse.

SQL> SELECT DISTINCT product_id FROM apr_orders WHERE product_id
NOT IN (SELECT product_id FROM product);

PRODUCT_ID
----------
SP1036

5.5.3 Looking up the warehouse key

Now that we have cleansed the PRODUCT_ID column, we will modify it to use the warehouse key. For the next example, a product_code has been added to the product table that will be used to look up the PRODUCT_ID, which is the surrogate key for the warehouse. Figure 5.4, showed theuse of surrogate keys in the warehouse. A portion of the product dimension is displayed.

SQL> SELECT PRODUCT_ID, PRODUCT_CODE FROM PRODUCT;

PRODUCT_ID PRODUCT_CODE
---------- --------
         1 SP1000
         2 SP1001
         3 SP1010
         4 SP1011
         5 SP1012

In this next transform we are going to use the PRODUCT_ID in the APR_ORDERS table to look up the warehouse key from the product dimension. The PRODUCT_ID column in the apr_orders table will be replaced with the warehouse key.

SQL> SELECT * FROM APR_ORDERS;

PRODUCT TIME_KEY  CUSTOMER  PURCHASE PURCH PURCH SHIP   TODAYS_
ID                ID        DATE     TIME  PRICE CHARGE SPECIAL
------- --------- --------- -------- ----- ----- ------ -------
SP1001 01-APR-02 AB123456   01-APR-02 24   28.01    6    Y
SP1001 01-APR-02 AB123457   01-APR-02 1024 28.01    6    Y
SP1061 01-APR-02 AB123456   01-APR-02   24 28.01    8.42 Y
SP1062 01-APR-02 AB123457   01-APR-02 1024 28.01    3.58 Y

SQL> UPDATE APR_ORDERS A SET A.PRODUCT_ID =
(SELECT P.PRODUCT_ID FROM PRODUCT P
WHERE A.PRODUCT_ID = P.PRODUCT_CODE);

4 rows updated.

Note that the original PRODUCT_ID's have been replaced with the warehouse key.

SQL> SELECT * FROM APR_ORDERS;

PRODUCT TIME_KEY  CUSTOMER  PURCHASE PURCH PURCH SHIP   TODAYS_
ID                ID        DATE     TIME  PRICE CHARGE SPECIAL
------- --------- --------- -------- ----- ----- ------ -------
2      01-APR-02 AB123456   01-APR-02 24   28.01    6    Y
2      01-APR-02 AB123457   01-APR-02 1024 28.01    6    Y
54     01-APR-02 AB123456   01-APR-02   24 28.01    8.42 Y
55     01-APR-02 AB123457   01-APR-02 1024 28.01    3.58 Y

5.5.4 Table functions

The results of one transformation are often stored in a database table. This table is then used as input to the next transformation. The process of transforming and storing intermediate results, which are used as input to the next transformation, is repeated for each transformation in the sequence.

The drawback to this technique is performance. The goal is to perform all transformations so that each record is read, transformed, and updated only once. Of course, there are times where this may not be possible, and the data must be passed through multiple times.

A table function is a function whose input is a set of rows and whose output is a set of rows, which could be a table-hence the name table function. The sets of rows can be processed in parallel, and the results of one function can be pipelined to the next before the transformation has been completed on all the rows in the set, eliminating the need to pass through the data multiple times.

Table functions use Oracle's object technology and user-defined data types. First, new data types must be defined for the input record and output table. In the following example, the PURCHASES_RECORD data type is defined to describe the records in the PURCHASES table.

SQL> CREATE TYPE purchases_record as OBJECT
(product_id                   VARCHAR2(8),
time_key                      DATE,
customer_id                   VARCHAR2(10),
purchase_date                 DATE,
purchase_time                 NUMBER(4,0),
purchase_price                NUMBER(6,2),
shipping_charge               NUMBER(5,2),
today_special_offer           VARCHAR2(1));

Next, the PURCHASES_TABLE data type is defined. It contains a collection of PURCHASES_RECORDs, which will be returned as output from the function.

SQL> CREATE TYPE purchases_table AS TABLE of purchases_record;

Next, define a type for a cursor variable, which will be used to pass a set of rows as input to the table function. Cursor variables are pointers, which hold the address of some item instead of the item itself. In PL/SQL, a pointer is created using the data type of REF. Therefore, a cursor variable has the data type of REF CURSOR. To create cursor variables, you first define a REF CURSOR type.

CREATE PACKAGE cur_pack AS TYPE ref_cur_type IS REF CURSOR;
END cur_pack;

Here is our table function , TRANSFORM, which performs the search and replace, removing the hyphen from the PRODUCT_ID and increasing the shipping charge. There are some things that differentiate it from other functions. The function uses PIPELINED in its definition and PIPE ROW in the body. This causes the function to return each row as it is completed, instead of waiting until all rows are processed. The input to the function is a cursor variable, INPUTRECS, which is of type REF_CUR_TYPE defined previously. The output of the function is a table of purchase records of type PURCHASES_TABLE defined previously. The REF CURSOR, INPUTRECS, is used to fetch the input rows, the transformation is performed, and the results for each row are piped out. The function ends with a RETURN statement, which does not specify any return value.

CREATE OR REPLACE FUNCTION transform (inputrecs
IN cur_pack.ref_cur_type)
RETURN purchases_table
PIPELINED
IS
  product_id                      VARCHAR2(8);
  time_key                        DATE;
  customer_id                     VARCHAR2(10);
  purchase_date                   DATE;
  purchase_time                   NUMBER(4,0);
  purchase_price                  NUMBER(6,2);
  shipping_charge                 NUMBER(5,2);
  today_special_offer             VARCHAR2(1);
BEGIN
LOOP
FETCH inputrecs INTO product_id,
      time_key,customer_id,purchase_date,purchase_time,
      purchase_price,shipping_charge,today_special_offer;
      EXIT WHEN INPUTRECS%NOTFOUND;
      product_id := REPLACE(product_id, '-','');
      shipping_charge :=(shipping_charge+shipping_charge*.10);
      PIPE ROW(purchases_record(product_id,
            time_key,
            customer_id,
            purchase_date,
            purchase_time,
            purchase_price,
            shipping_charge,
            today_special_offer));
END LOOP;
CLOSE inputrecs;
RETURN;
END;

We've rolled back the changes from the previous transforms and are going to do them again using the table function. Here are the data prior to being transformed.

SQL> SELECT * FROM APR_ORDERS;

PRODUCT TIME_KEY  CUSTOMER  PURCHASE PURCH PURCH SHIP   TODAYS_
ID                ID        DATE     TIME  PRICE CHARGE SPECIAL
------- --------- --------- -------- ----- ----- ------ -------
SP-1001 01-APR-02 AB123456   01-APR-02   24 28.01   5.45  Y
SP-1001 01-APR-02 AB123457   01-APR-02 1024 28.01   5.45  Y
SP1061  01-APR-02 AB123456   01-APR-02   24 28.01   8.42  Y
SP1062  01-APR-02 AB123457   01-APR-02 1024 28.01   3.58  Y

To invoke the function, use it as part of a SELECT statement. The TABLE keyword is used before the function name in the FROM clause. The changes are shown in bold in the following code:

SQL> SELECT * FROM
     TABLE(transform(CURSOR(SELECT * FROM apr_orders)));

PRODUCT TIME_KEY  CUSTOMER  PURCHASE PURCH PURCH SHIP   TODAYS_
ID                ID        DATE     TIME  PRICE CHARGE SPECIAL
------- --------- --------- -------- ----- ----- ------ -------
SP1001  01-APR-02 AB123456  01-APR-02   24 28.01    6       Y
SP1001  01-APR-02 AB123457  01-APR-02 1024 28.01    6       Y
SP1061  01-APR-02 AB123456  01-APR-02   24 28.01    9.26    Y
SP1062  01-APR-02 AB123457  01-APR-02 1024 28.01    3.94    Y

If we needed to save the data, the following code creates a table to save the results of the table function.

SQL> CREATE TABLE TEST AS
SELECT * FROM TABLE(transform(CURSOR(SELECT * FROM apr_orders)));
Table created.

5.5.5 Transformations that split one data source into multiple targets

Sometimes transformations involve splitting a data source into multiple targets, as illustrated in Figure 5.15. A new feature introduced in Oracle 9i, multitable INSERT, facilitates this type of transformation.

Click To expand
Figure 5.15: Multitable insert.

In the following example, the APR_ORDERS will be split into two tables. All orders that took advantage of today's special offer will be written to the SPECIAL_PURCHASES table. All regular sales will be written to the PURCHASES table. This information could be used to target future advertising. A new table, SPECIAL_PURCHASES, has been created with the same columns as the APR_ORDERS and PURCHASES tables.

The insert statement specifies a condition, which is evaluated to determine into which table each row should be inserted. In this example there is only one WHEN clause, but you can have multiple WHEN clauses if there are multiple conditions to evaluate. If no WHEN clause evaluates to true, the ELSE clause is executed.

By specifying FIRST, Oracle stops evaluating the WHEN clause when the first condition is met. Alternatively, if ALL is specified, all conditions will be checked for each row. ALL is useful when the same row is stored in multiple tables.

SQL> INSERT FIRST WHEN today_special_offer = 'Y'
     THEN INTO special_purchases
     ELSE INTO purchases
     SELECT * FROM apr_orders;

15004 rows created.

The four rows with today_special_offer = 'Y', have been inserted into the SPECIAL_PURCHASES table. The remaining rows have been inserted into the PURCHASES table.

SQL> SELECT COUNT(*) FROM purchases;

  COUNT(*)
----------
    15000

SQL> SELECT COUNT(*) FROM special_purchases;

   COUNT(*)
----------
      4

From the following queries, we can see that the data has been split between the two tables. Purchases made with the value of N in TODAY_SPECIAL_OFFER are stored in the PURCHASES table. Those with the value of Y are stored in the SPECIAL_PURCHASES table.

SQL> SELECT DISTINCT(today_special_offer) FROM purchases;

Today_special_offer
-------------------
N

SQL> SELECT DISTINCT(today_special_offer) FROM special_purchases;

Today_special_offer
-------------------
Y

5.5.6 Moving data from a staging table into the fact table

Once the data has been transformed, it is ready to be moved to the warehouse tables. In Figure 5.14, the April purchases were moved into temporary staging tables in the warehouse. Once the data has been cleansed and transformed, it is ready to be moved into the purchases fact table, as shown in Figure 5.16.

Click To expand
Figure 5.16: Data is moved from a staging table into the fact table.

When the fact table is partitioned, new tablespaces are created for the data file and indexes, a new partition is added to the fact table, and the data is moved into the new partition. The steps are illustrated using the EASYDW example.

Step 1: Create a new tablespace for the April purchases and the April purchases index

Since the purchases table is partitioned by month, a new tablespace will be created to store the April purchases. Another tablespace is created for the indexes.

SQL> CREATE TABLESPACE purchases_apr2002
     DATAFILE 'c:\ora9ir2\oradata\orcl\purchasesapr2002.f'
     SIZE 5M
     REUSE AUTOEXTEND ON
     DEFAULT STORAGE
     (INITIAL 64K NEXT 64K PCTINCREASE 0 MAXEXTENTS UNLIMITED)

SQL> CREATE TABLESPACE purchases_apr2002_idx
     DATAFILE 'C:\ora9ir2\oradata\orcl\PURCHASESAPR2002_IDX.f'
     SIZE 3M REUSE
     AUTOEXTEND ON
     DEFAULT STORAGE
     (INITIAL 16K NEXT 16K PCTINCREASE 0 MAXEXTENTS UNLIMITED)
Step 2: Add a partition to the purchases table

The PURCHASES table is altered, and the purchases_apr2002 partition is added.

SQL>ALTER TABLE easydw.purchases
ADD PARTITION purchases_apr2002
VALUES LESS THAN (TO_DATE('30-04-2002', 'DD-MM-YYYY'))
PCTFREE 0 PCTUSED 99
STORAGE (INITIAL 64K NEXT 164 PCTINCREASE 0)
TABLESPACE purchases_apr2002;

Figure 5.17 shows the new partition that has been added to the PURCHASES table and the corresponding PURCHASE_PRODUCT_INDEX.

Click To expand
Figure 5.17: Viewing a table's partitions using enterprise manager.
Step 3: Move the table into the new partition

There are a variety of ways to move the data from one table to another in the same database:

  • Exchange partition (when the fact table is partitioned)

  • Direct path insert

  • CREATE TABLE AS SELECT

Moving data using exchange partition

The ALTER TABLE EXCHANGE PARTITION clause is generally the fastest way to move the data of a nonpartitioned table into a partition of a partitioned table. This clause can be used to move both the data and local indexes from a staging table into a partitioned fact table. The reason it is so fast is because the data is not actually moved; instead, the metadata is updated to reflect the changes.

If the data has not previously been cleansed, it can be validated to ensure that it meets the partitioning criteria, or this step can be skipped using the WITHOUT VALIDATION clause. Figure 5.18 shows moving the data from the APR_ORDERS staging table into the PURCHASES fact table using exchange partition.

Click To expand
Figure 5.18: Exchange partition.

The following example shows moving the data from the APR_ORDERS into the PURCHASES_APR2002 partition of the EASYDW.PURCHASES table.

SQL> ALTER TABLE easydw.purchases EXCHANGE PARTITION purchases_
apr2002 WITH TABLE apr_orders WITHOUT VALIDATION;

After exchanging the partition, there are no rows left in the APR_ORDERS table, and it can be dropped.

SQL> SELECT * FROM apr_orders;

no rows selected

SQL> DROP TABLE apr_orders;

Table dropped.

Moving data between tables using direct path insert

If the fact table is not partitioned, you can add more data to it by using direct path insert. A direct path insert enhances performance during insert operations by formatting and writing data directly into the data files without using the buffer cache. This functionality is similar to SQL*Loader direct path mode.

Direct path insert appends the inserted data after existing data in a table; free space within the existing table is not reused when executing direct path operations. Data can be inserted into partitioned or nonpartitioned tables, either in parallel or serially. Direct path INSERT updates the indexes of the table.

In the EASYDW database, since the purchases table already exists and is partitioned by month, we could use the direct path insert to move the data into the table. Direct path insert is executed when you include the APPEND hint and are using the SELECT syntax of the INSERT statement.

SQL>INSERT /*+ APPEND */ INTO easydw.purchases
          SELECT * FROM apr_orders

In our example, the purchases fact table already existed, and new data was added into a separate partition.

Hint:

Be sure you have disabled all references constraints before executing the direct load insert. If you do not, the APPEND hint will be ignored, no warnings will be issued, and a conventional insert will be used. Plus, the insert will take a long time if there is a lot of data.
Hint:

Conventional path is used when using the INSERT... with the VALUES clause, even if you use the APPEND hint.

Creating a new table using CREATE TABLE AS SELECT

If the detail fact table does not yet exist, you can create a new table selecting a subset of one or more tables using the CREATE TABLE AS SELECT statement. The table creation can be done in parallel. You can also disable logging of redo. In the following example, TEMP_PRODUCTS is the name of the staging table. After performing the transformations and data cleansing, the products table is created by copying the data from TEMP_PRODUCTS table.

SQL> CREATE TABLE products PARALLEL NOLOGGING AS
           SELECT * FROM temp_products;


Postload operations

5.6 Postload operations

After the data is loaded, you may need to validate its quality, reenable constraints and triggers, rebuild indexes, update the cost-based optimizer statistics, and refresh the materialized views prior to making the data available to the warehouse users.

Step 1: Gather optimizer statistics for the tables

Run the dbms_stats package to update the optimizer statistics on any tables where you have added a significant amount of data. Statistics can be gathered for an index, index partition, table, partition or subpartition, and a materialized view.

Since we added an entire partition to the purchases fact table, the following example will gather statistics for the purchases_apr2002 partition.

SQL> EXECUTE DBMS_STATS.GATHER_TABLE_STATS('easydw','purchases',
'purchases_apr2002',DBMS_STATS.AUTO_SAMPLE_SIZE);
Step 2: Verify the dimensions

If using Summary Management, run the DBMS_OLAP.VALIDATE_DIMENSION procedure for each dimension to verify that the hierarchical, attribute, and join relationships are correct. It can also be used to determine if any level columns in a dimension are NULL. You can either verify the newly added rows or all the rows. Any exceptions are logged in the table MVIEW$_EXCEPTIONS, which is created in the user's schema.

In the following example, the product dimension (parameter 1) in the EASYDW schema (parameter 2) is verified for correctness. All rows will be validated (parameter 3 is set to false), and we'll check for nulls (parameter 4 is set to true).

SQL> EXECUTE dbms_olap.validate_
dimension('product','easydw',false,true);

SQL> select * from mview$_exceptions;
no rows selected
Step 3: Refresh the materialized views

After loading the April data into the purchases table, the PRODUCT_SUM materialized view became stale, as shown in Figure 5.19. It only contains information on the sales for January through March and must be refreshed to add April's data.

Click To expand
Figure 5.19: Using Oracle Enterprise Manager to look for stale materialized views.

Figure 5.20 shows various options you can choose when refreshing your materialized views. In this case we've chosen "force" for the type of refresh, which will incrementally refresh the materialized view if possible. If it cannot be incrementally refreshed, a complete refresh will be done. We have chosen to refresh on demand. The refresh could also have been scheduled to run at a specific time. If data is being updated in the warehouse instead of being bulk loaded, it is also possible to refresh the materialized views after committing each transaction, which would cause them to go stale. A materialized view log must be created to be able to refresh upon commit.

Click To expand
Figure 5.20: Refresh options.

By pressing the Explain button, you can determine the capabilities of your materialized view, including what type of refresh the system will be able to perform. Figure 5.21 shows that this materialized view will be completely refreshed. In Chapter 4, partition change tracking (PCT) was described. PCT allows materialized views on partitioned tables to be fast refreshed after partition maintenance operations. In order to use PCT, either the partitioning key for the detail tables must be included in the materialized view or you can include a special column known as the partition marker. In this example, it cannot use PCT, because there is no partition key or pmarker in the select list. Another reason it is not fast refreshable is that some columns are missing from the materialized view definition. We can see there is a SUM(expr) without a COUNT(expr). It cannot be fast refreshed after an insert or other DML statement, since it doesn't have a materialized view log. In order to use these optimizations, the materialized view definition needs to be modified. Explain materialized view can be a big help in determining how to best define your materialized views.

Click To expand
Figure 5.21: Explain materialized view.

An alternate way to refresh the materialized views is by executing the DBMS_MVIEW.REFRESH procedure, as previously discussed in Chapter 4.

Step 4: Gather optimizer statistics for the materialized views

Next, update the optimizer statistics on any materialized views that may have significantly changed in size. If you used Oracle Enterprise Manager to refresh your materialized views, you have the option of analyzing the materialized view after it was refreshed, as shown in Figure 5.22.

Click To expand
Figure 5.22: Analyzing the materialized view after refresh.

To gather optimizer statistics outside of Oracle Enterprise Manager, use the DBMS_STATS package. In this example table statistics will be gathered from the PRODUCT_SUM materialized view in the EASYDW schema.

SQL> EXECUTE DBMS_STATS.GATHER_TABLE_STATS('easydw','product_sum');
Step 5: Back up the database, table, or partition after the load is complete (if you used the UNRECOVERABLE option)

Since media recovery is disabled for the table being loaded, you will not be able to recover in the event of media failure. Refer to Chapter 8 for a discussion on backup techniques.

Step 6: Publish the data

Send e-mail to notify the users of what data has been loaded the previous day and is ready for their use.


Using tools for the ETL process

5.7 Using tools for the ETL process

Several tools are available in the marketplace to help automate parts of the ETL process, including Oracle Warehouse Builder, Informatica, AbInitio, Ascential, and Sagent. These tools provide the mechanisms to connect to the heterogeneous data sources, generally to either relational databases or flat files, and perform the data extraction functions. They control the transfer of data across a network, provide data transformation services and data cleansing capabilities, and load the data into your data warehouse.


Summary

5.8 Summary

In this chapter we looked at the ETL process. Various techniques were discussed to identify rows that changed on the source system as part of the extraction process, and Oracle change data capture was introduced. A number of types of transformations that are common to a data warehouse were discussed. Examples were shown performing transformations during the load process and using staging tables inside the Oracle database. Techniques used to load the warehouse were described including SQL*Loader, external tables, and transportable tablespaces. Moving data from the staging system to the warehouse tables can be done using exchange partition, direct path insert, or CREATE TABLE AS SELECT. In Chapter 6, we'll look at Oracle Warehouse Builder, which uses many of the functions discussed in this chapter to help you load your data warehouse.


Chapter 6: Oracle Warehousing Tools

6.1 Which tool?

So far we have built and managed our database using the SQL*Plus interface, but there are a number of tools available to the user, DBA, and designer to facilitate easy warehouse creation and access. Now we will review some of the tools available from Oracle, which includes:

  • Warehouse Builder

  • Discoverer

  • Reports (described in Chapter 8)

  • OLAP and data mining (described in Chapter 9)

Oracle Warehouse Builder is a tool that helps the DBA design and manage the data warehouse, whereas Oracle Discoverer is an end-user tool for querying your data warehouse.


Oracle Warehouse Builder

6.2 Oracle Warehouse Builder

Oracle Warehouse Builder (OWB) is Oracle's tool for designing and deploying data warehouses, data marts, and business intelligence applications. It is part of the Oracle 9i Developer Suite, which includes products for application development, such as 9i Jdeveloper, 9i Designer, and 9i Forms Developer. The Business Intelligence area includes products such as Oracle Discoverer (described later in this chapter), Oracle Reports (described in Chapter 8), and Oracle Warehouse Builder.

Building any type of application is not a task to be undertaken lightly, and the many steps to be completed when building our data warehouse and BI application can be facilitated by Oracle Warehouse Builder. It can:

  • Design and create the data flows between sources and targets

  • Design, create, manage, update, and upgrade the data warehouse schema

  • Manage and update the source definitions

  • Import data source definitions

  • Design and create the OLAP and ad hoc query environment

  • Take advantage of all Oracle database features

Within OWB you will find a repository stored in an Oracle database, and this is where OWB keeps all of its metadata. There is the OWB client, which is used to design and create the data warehouse and application. A code generator is provided to create the scripts from your design that are applied to the data warehouse, and there is a reporting environment so that you can view your metadata from a browser.

You may be wondering why you should use a tool such as Warehouse Builder to design and create your data warehouse and associated application, why not design it by hand? Yes, you could do that, but some of the benefits of using Warehouse Builder include:

  • System design time is reduced, due to the GUI.

  • The design is held in one place, so users are guaranteed that they are not working with an out-of-date model.

  • The code generated by OWB is free from errors and works the first time.

Let us now look at how we would use OWB to design our data warehouse or data mart.

6.2.1 Setting up Warehouse Builder

Before your data warehouse can be designed using Warehouse Builder, a small amount of setup is required to create the repository that Warehouse Builder uses when designing your warehouse. Launching the Repository Assistant, the screen shown in Figure 6.1 shows us the steps that have to be completed to set up our Warehouse Builder Repository. Although there are quite a few steps, this entire process does not take very long, so you will soon be up and running Warehouse Builder.

Click To expand
Figure 6.1: Warehouse Builder—steps to setup.

You can choose in which database the OWB repository will reside, and then you must connect using a user name that has the SYSDBA privilege. A user name and password must be supplied for the OWB repository, as well as which tablespace to use for repository data and which language you would like to use. Click on Finish and your repository is created; now Oracle Warehouse Builder is ready to use.

6.2.2 Oracle Warehouse Builder client

Once the Oracle Warehouse Builder repository has been built, it is time to start using this tool to design our data warehouse, where we can:

  • Define the logical view of our warehouse schema

  • Define data sources and targets

  • Describe ETL processes, where we extract, transform, and load

  • Generate the SQL required to create the data warehouse

  • Schedule loading and refresh jobs using Oracle Enterprise Manager

In OWB, your data warehouse is defined inside a project and within that project there is what is known as a set of formal definitions defining how the warehouse looks from a logical standpoint. These definitions are referred to as modules.

The steps we need to go through to create our data warehouse include, defining the source and target databases. We then need to describe how the data will be moved from those sources to our data warehouse. In moving the data, we will need to describe transformations to get the data into the format we require inside the warehouse. Then we have to generate the design and code to implement and load our warehouse.

6.2.3 Creating modules

One of the first steps is to define where the data for our warehouse will originate. This is achieved by defining a module using the New Module Wizard; this requires the completion of four steps. One of the first steps is shown in Figure 6.2, where we give our module a name, Order_Entry_system. Then from a drop-down list we can choose our module's status and, since we are defining the source of our data, we have selected production.

Click To expand
Figure 6.2: Warehouse Builder—creating a module.
Hint:

Spaces in the module name are not allowed if you are defining a physical object.

OWB provides several sources for our data, including flat files and SAP. In Figure 6.3, we have selected an Oracle database.

Click To expand
Figure 6.3: Warehouse Builder—type of data source.

Since we specified that one of our data sources is an Oracle database, we must now determine whether it is an Oracle database or if we wish to take our design from the repository in Oracle Designer. Then all we have to do is create a dblink to our database using the supplied screen, as shown in Figure 6.4, and our data source is now defined.

Click To expand
Figure 6.4: Warehouse Builder—data source defined.

This process can then be repeated again to specify where our data warehouse is to be located. In Figure 6.5, we can see the main navigator from OWB Client, showing that we have two modules created: EASYDW_DW, which is our data warehouse, and ORDER_ENTRY_SYSTEM, which is one of the sources of data for our warehouse. Another useful feature is that since we are actually building the data warehouse, OWB gives us the ability to say what type of data warehouse we require and that at this time it does not have to physically exist.

Figure 6.5: Warehouse Builder—two modules created.

After doing all of this work it is extremely important that your work is saved, because OWB does not automatically save your work. Therefore, when you are satisfied with the state of the objects, you must commit the changes by either clicking on the Commit icon or selecting Commit from the File option on the strip menu at the top.

6.2.4 Defining the tables in our data warehouse

In the previous section we defined the sources for our data and our data warehouse. Now is the time to define the tables that we require inside our data warehouse. This can be achieved by either defining the tables manually or importing them from another database.

Importing the table definitions illustrates how OWB can save on development time, because it can be quite time consuming defining the tables by hand and it is also error prone. When you are defining these tables, which are going to be the sources for your data, it's very important that they are defined exactly as they exist in your production systems. If you have to define them by hand, then it's easy to make mistakes, which can subsequently delay implementation of your data warehouse. The table definitions are imported using the Import Metadata Wizard, which is shown in Figure 6.6, where we have selected some tables from another database.

Click To expand
Figure 6.6: Warehouse Builder—import a table.

Some of the tables may have to be defined manually, and this can be achieved using the New Table Wizard. Simply double-click on the module, EASYDW_DW, as shown in Figure 6.5. This will start the new window, shown in Figure 6.7, which is the Warehouse Module Editor. Here, select Table and then click using the right mouse button; the Create Table option will appear.

Click To expand
Figure 6.7: Warehouse Builder—manually define a table.

Now we can define our table manually by specifying all of the columns, as shown in Figure 6.8. While using this wizard, don't forget that you can also specify constraints, such as primary key, foreign key, and check constraints. When the definition is complete, click on the Finish button to complete the definition of our table.

Click To expand
Figure 6.8: Warehouse Builder—Warehouse Module Editor.

At this time we have only shown how to create a table (see Figure 6.7), which shows the Warehouse Module Editor. Here you can also define materialized views, which were described in Chapter 4.

6.2.5 Creating dimensions

Typically a data warehouse is comprised of tables, which can include fact and dimension tables. At this stage in the development of our data warehouse, we probably only have a model of our data warehouse, described in terms of facts and dimensions. OWB allows us to define these logical objects and then when it is time to generate our design, OWB will physically create the tables to represent these facts and dimensions.

A dimension is created in OWB by using the Dimension Wizard, which is selected from the Warehouse Module Editor by right-clicking on Dimensions and then selecting either Create Dimension or Create Time Dimension.

The process is very similar to the one we saw in Chapter 4 for creating a dimension. That is, you must name the dimension, define each level and its attributes, and then describe the hierarchy. In Figure 6.9, we see one of the Dimension Wizard screens, where we are defining the hierarchy for our customer dimension.

Click To expand
Figure 6.9: Warehouse Builder—dimension hierarchy.

If you want to create a time dimension, OWB will automatically suggest appropriate levels, attributes, and hierarchies, and then you can choose what is appropriate for your system. This means that time dimensions can be created very quickly and easily. OWB also provides a PL/SQL routine, which will insert data into the time dimension; it can also be configured to generate an arbitrary set of dates.

6.2.6 Creating fact tables

Once the dimensions have been created, the next step in our design process is to define one or more fact tables. In OWB a fact table is a logical object in that only when the physical design is generated, does our fact table become an actual physical table. It is created by right-clicking on Fact in the Warehouse Module Editor and selecting Create Fact; the New Fact Wizard now appears. One of the screens is shown in Figure 6.10, which is where the foreign keys for our fact table are defined. In this example, OWB automatically offers a foreign key to each dimension that was previously defined.

Click To expand
Figure 6.10: Warehouse Builder—fact table creation.

The wizard also allows you to add measures, and by clicking on the Finish button our fact table is created. We can now see the dimensions and facts in Figure 6.11 in the Warehouse Fact Table Editor.

Click To expand
Figure 6.11: Warehouse Builder—Fact Table Editor.
Hint:

Validate your design and individual components frequently.

At any time while the data warehouse is being defined, the various parts of the design can be validated by selecting the Validate option, described in section 6.2.9. Any problems that are identified can be corrected immediately before proceeding to defining the next part of the design.

6.2.7 Transformations

There is more to the design of our data warehouse than simply defining a few tables. We are using OWB because we want to be able to easily define how data is to be moved from our source database into our data warehouse. We saw in Chapter 5 how data, while it is being moved from one source to another, often needs to undergo some transformations. In OWB we can define these transformations using:

  • Procedures

  • Package procedures

  • PL/SQL functions

  • Package functions

Once again, there should be no need for us to write anything new, because OWB comes with a number of supplied transformations, which should match your requirements. In Figure 6.12, we can see some of the supplied functions. OWB groups the functions together, and here we can see some of the date functions that are available, such as ADD_MONTHS or find the LAST_DAY.

Click To expand
Figure 6.12: Supplied functions for transformations.

In the unlikely event that there is nothing suitable in the supplied libraries, you can easily create your own functions and procedures within Oracle Warehouse Builder; but hopefully this should not be necessary. Now that we know these functions are present in OWB, we can start to define how we move our data from our source databases into our data warehouse.

6.2.8 Defining source to target mappings

The real power of Oracle Warehouse Builder starts to become apparent when we see how it can be used to define how data is moved from our sources, such as our OLTP system, into our data warehouse. In Chapter 5 we learned of the various techniques we can use to load and perform transformations. Now, inside Oracle Warehouse Builder, using its GUI interface, we can graphically represent these processes by defining a mapping. When the design is finally generated, OWB even creates the procedures required to transform and load the data. Now we will look at just a few of the many different types of mappings that are possible.

Mapping a source to a target

A mapping is defined from the Warehouse Module Editor by right-clicking on Mappings and selecting Creating Mappings, the Mapping Editor appears, as shown in Figure 6.13. The first mapping we are going to define is extracting information from our ORDERS OLTP system and moving it into our data warehouse. OWB can also take information from flat files or from SAP.

Click To expand
Figure 6.13: Mapping Editor.

When the blank Mapping Editor appears, click on the Mapping Table icon and drag it onto the Editor. It now asks you where your table is to come from, and we are going to select the ORDER_LINES table from our source ORDER_ENTRY_SYSTEM, which we defined earlier. Then click on the Mapping Fact icon and select PURCHASES_FACT. In our editor we now have two tables: ORDER_ITEMS and PURCHASES_FACT.

Now we must specify which items are to be moved from each table. First we are going to move the item PRODUCT_ID from the ORDER_ITEMS table to the PURCHASES_FACT table. This is achieved by dragging a line from product_id in ORDER-ITEMS to product_id in PURCHASES_ FACT. In Figure 6.13 we can see the line that OWB has drawn between the two items.

Computing a value

The next column to be defined in PURCHASES_FACT is the value of the order, and this can only be obtained by computing its value from two columns in the ORDER_ITEMS table. To compute this value an intermediate expression box must be created, as illustrated in Figure 6.13.

Click on the Expression icon and drag that onto the Mapping Editor. An empty box appears with empty input and output groups. Take the two columns from ORDER_ITEMS, unit_price and quantity, and drag them over to the input group box. Now we have to use the right mouse button to add a new attribute in the output group, which we will call Total_Price. Then select that item's properties and click on the Expression box and the Expression Builder. Here you can specify how the attribute is to be computed, which in our case is to multiple the two numbers together. Then drag a line from Total_Price to the column Sum_Purchase in PURCHASES_FACT to complete the mapping operation.

Hint:

Its probably worth validating periodically so that you don't create mappings and transformations that are invalid.

Joining tables to obtain data

Another common task that our warehouse designer has to perform is joining data from two or more tables to extract information that is used as input to another table. In Figure 6.14, we see a join that we have created between ORDERS and ORDER_ITEMS to enable us to store one record in our data warehouse about the total value of the order. Note that the expression we created in Figure 6.13 has now been used as input to the join.

Click To expand
Figure 6.14: Mapping Editor with joins.
Hint:

When you have finished working on a part of the design, such as the expression to compute the value of the sales, you can minimize the box to give you more working space.

Key lookup

Previously in this book we have described surrogate keys, where the keys used in the sources for your data warehouse are transformed into a key used by the data warehouse. In OWB we can define exactly how that transformation should occur using the key lookup feature.

In Chapter 5 we described the process of converting the product code used in out OLTP system to a surrogate key in the data warehouse. In Figure 6.15, this has been implemented in OWB by showing that the column product_id in ORDER_ITEMS is input to the key lookup process.

Click To expand
Figure 6.15: Product key lookup.

Although not visible, a mapping has been defined that states that product_ id is to be matched to product_cd. The resulting output is the column product_id, which is sent to the PURCHASES_FACT table.

Filtering data

When data is being extracted from sources such as our OLTP system, there may be times when we do not want all of the data to be sent across to our data warehouse. In OWB this is not a problem, because it allows us to filter the incoming data in a variety of ways.

In Figure 6.16, we are filtering the items from the ORDER_ITEMS table according to a criterion that we have specified. For example, we could say that we are only interested in data where the total value of the order is greater than a specified amount. Also, after we have filtered the data, we then apply the expression we defined in Figure 6.13 to the data that have passed through the filter.

Click To expand
Figure 6.16: Filtering source data.

Using data generators

There may be times when, rather than extracting data from a source, this data must be automatically generated. OWB has this capability, and it can automatically create:

  • Record number

  • Sequence number

  • System date

In Figure 6.16, we can see the system date being used to set the date for our order. This was achieved by dragging the Data Generator icon onto the Mapping Editor and then selecting the item required and attaching it to the fact table.

So far in this chapter, we have barely touched the surface on the types of transformations and mappings that are possible with Oracle Warehouse Builder. We have kept these examples simple, but in a real warehouse they would be connected together. We started to show this in Figures 6.14 and 6.16, where the expression was being used as input to another stage in the loading process.

6.2.9 Validating the design

Once the design is complete, or while it is being developed, it must be validated before Oracle Warehouse Builder can create all the components needed to build, load, and manage our data warehouse. You can validate each component individually or validate the entire project. It's probably easier to resolve problems if you validate each component as it is defined.

To validate any component simply select Validate from the list of menu options. In Figure 6.17, we see that when we validated our key lookup, there was a problem with the column we are comparing against not being unique. We can now fix this problem by making the column PRODUCT_CD in the PRODUCT table unique; then this module will validate without errors.

Click To expand
Figure 6.17: Validating the design.

While the design is being validated, Oracle Warehouse Builder displays its progress, as shown in Figure 6.18. This is extremely useful when you are validating the entire design. It actually has quite a lot of work to do, and it's nice to see how it is progressing; otherwise, you might think that the validation process has stalled.

Click To expand
Figure 6.18: Validation progress.

6.2.10 Generating and configuring the design

With our design validated, we are now ready to ask Oracle Warehouse Builder to generate our design. From the Warehouse Module Editor, shown in Figure 6.7, we can generate each component individually by selecting Module and then Generate. OWB will ask if we want to create some scripts it is capable of generating:

  • SQL for creating the database

  • PL/SQL for executing within the database

  • Procedures for loading the data

  • SQL*Loader files for working with our flat files

  • TCL for Oracle Enterprise Manager

In Figure 6.19 we can see the various components that OWB generates for us. There are the schema objects, such as our PRODUCT table. By clicking on the button View Code, we can see the table definition; alternatively, we can click on Save as File so we can implement it later. If we decide that we want to do it now, then click on the Deploy button, OWB will ask us into which database this is to be created and then it will create it there.

Click To expand
Figure 6.19: Implementing the design.

If we click on the Mappings tab, we would see that OWB has created a package for us to load the data we defined in Figure 6.14. Clicking on the View Code button, we can see part of this long procedure generated by OWB in Figure 6.20. Just look at where the slider bar is to see how much code OWB has created. The section we have shown here is part of the INSERT statement, but prior to this there are variable definitions and lots of other things that we would have had to write ourselves.

Click To expand
Figure 6.20: Code generated by Oracle Warehouse Builder.

In this example OWB has chosen to use a SQL INSERT statement, but since OWB has been designed to take advantage of features in the database, it can use statements such as MERGE, which we saw in Chapter 3 in the code that it generates.

Although we have generated a design, it is most likely that it does not include all of the physical aspects of our design, such as whether a table is partitioned and what indexes are needed. These physical components can be configured in OWB by selecting the module to be configured; from Edit on the menu, select Configure. Figure 6.21 lists all of the properties you can configure. In this example for our PURCHASES_FACT table, we can see that foreign-key constraints already exist; the table is going to be stored in the tablespace, USERS, and we haven't yet partitioned the table.

Click To expand
Figure 6.21: Configuring the physical design.

In the Indexes section you will see that three bitmapped indexes are present. These have been automatically configured by OWB, as illustrated in Figure 6.22; by clicking on the Add button for an index, Generate is pressed and it recommends bitmapped indexes on the foreign-key columns.

Click To expand
Figure 6.22: Bitmapped index recommendations.

Hopefully you now have an appreciation of what OWB can do. This is a very powerful product, and we have only been able to highlight some of its features. There are many benefits from using tools such as Oracle Warehouse Builder. You can save huge amounts of time, because you no longer have to write the code needed to implement and load data into your warehouse. The code that OWB generates is optimized for the Oracle database and it works immediately. It also provides a visual representation of your warehouse, and by using the various wizards that are available, it is easy to complete the tasks needed for building the warehouse. In addition, when changes to the environment occur, you can visually see the impact of them; OWB can easily incorporate them into the environment. Finally, even if you prefer to design your warehouse by hand, there are bound to be some of the monotonous tasks associated with warehouse design that OWB can complete quickly and accurately.


Oracle Discoverer

6.3 Oracle Discoverer

Oracle Discoverer is an extremely popular tool for querying data warehouses and generating reports, because it is very easy and intuitive to use and has been designed for use by users who are not computer literate. Therefore, in order for our non-computer-literate users to be able to use Discoverer easily, some setup is required from the DBA group. But once this has been done, your end users should really like using this tool. There are four parts to Oracle Discoverer:

  • Administrator

  • Desktop

  • Plus

  • Viewer

Discoverer Administrator is the version that is used by the DBAs of the data warehouse to set up the environment for the general Discoverer users. General users will either use the Desktop or Plus versions, because they have been designed especially for people who are not familiar with writing computer programs, as well as for anyone who is not familiar with SQL and prefers to deal with data using familiar business entities. For those users who may be concerned that they could change the data within the data warehouse, fear not; Discoverer Viewer allows the user only to view predefined reports. The attraction of using Discoverer Viewer is that all users require to access the data is their PC and a browser.

Oracle Discoverer is also integrated with Oracle Portal, which we will learn more about in Chapter 8. Via this route, access to data can be achieved via a list of workbooks or a worksheet/graph portal.

The use of Oracle Discoverer is not restricted to Oracle databases, but that is how we will illustrate its use here. The databases that have been certified against Discoverer include DB2 for NT and MVS, Microsoft SQL Server, Sybase, and Terradata.

6.3.1 Why Discoverer?

Before we learn how to set up Discoverer, let's first look at the types of reports that it can produce.

Query using Discoverer Viewer

Discoverer is a very powerful reporting tool; there is a version called Discoverer Viewer, which provides a user with the ability to access predefined reports and have the capability to modify those reports in a limited way. The advantage of this approach is that the user only needs a PC and a browser to access the data in our warehouse; the DBA can rest assured that users cannot change the data.

You can start Discoverer viewer from your browser using a URL such as:

http://easydw.com:7777/discoverer/viewer

A connection is made to this database, and a list of the available workbooks to which we can attach is presented. We select our workbook, Easydw_workbook, and the query contained within is immediately executed. In Figure 6.23 we can monitor the progress of our executing query; when it has completed, our report is displayed-as shown in Figure 6.24.

Click To expand
Figure 6.23: Discoverer Viewer-executing our query.
Click To expand
Figure 6.24: Discoverer Viewer-category sales by month report.

Since we defined this report as both a table and a graph, both versions are displayed. Note that at any time we can hide the graph or table by clicking on the Hide Data or Hide Chart option. The report shown here is the default generated by Discoverer; I am sure you will agree that it is a very nice format which is presented well and is easy to understand.

Inside a workbook there can be many reports, and each one is accessed by simply clicking on the tab on the left-hand side where the report is named.

One of the advantages of using Discoverer Viewer is that the user is not able to change the data in the warehouse or generate new reports. However, there are limited customizations that users can do to this report, such as changing the sort order or table layout or by clicking on the Presentation Options items, as illustrated in Figure 6.25, to change how the report looks.

Click To expand
Figure 6.25: Discoverer Viewer-presentations options (screen 1).

Dynamic reports with graphs

One of the features that makes Discoverer nice to use is the ability to group together a number of reports that can be easily run and customized to your own requirements. In the example shown here, we have a workbook called Easydw_reports, and, referring to Figure 6.26, at the bottom of the screen we can see three tabs for each of the reports that are available:

  • Total Sales by Customer

  • Best Customers

  • Most Popular Products

Click To expand
Figure 6.26: Discoverer Viewer-presentations options (screen 2).

In Figure 6.26, we see both a table on the left and a graph on the right generated from data in our warehouse. When this report is run, users are prompted to specify their criteria for what makes a best customer (e.g., purchases more than $15,000) and then the data is displayed. So we can see that our best customer at the moment is a company called Harry Travel.

Drilling on the data

When a report is produced, people reading it may feel it is very interesting, but they need to know more about how this data is derived. For example, suppose a user is viewing the report shown in Figure 6.27 and sees that cameras were the most frequently sold item. This could raise the question: Do we sell the same number of cameras throughout the year or is it seasonal?

Click To expand
Figure 6.27: Discoverer Viewer-presentations options (screen 3).

Anyone viewing this report can right-click on an item, such as camera, and select the Drill option. Discoverer will now ask you how you want to drill on this data. In Figure 6.27, we have drilled down to see the sales by month, and we see that we are selling about the same number of cameras every month. When drilling on the data, you only have to drill on the items you are interested in.

Now that we have caught a glimpse of the types of reports Discoverer can produce, we must return to setting up Discoverer; before you can use it, the setup we mentioned earlier must be performed.

6.3.2 Setting up the environment

Although this may seem like a lot of work initially, once completed, users will really appreciate the work you have done. The environment that you create will determine exactly which data in the warehouse your users can see and how they see this data. Most important, they won't need to understand how to join data in order to query the data warehouse. To begin using Discoverer, first connect to your Oracle database.

Hint:

You may prefer to create a special user for Discoverer and connect to your database using that user to ensure that the metadata is loaded under that user name.

End-user layer

The first time the Discoverer Administrator is started, you will be asked to create an end-user layer (EUL), which comprises all the metadata that is needed by Discoverer; therefore, this task will be performed only once. Figure 6.28 shows the screen where you can manage the end-user layer, including creating a new one or deleting an existing one.

Click To expand
Figure 6.28: Discoverer Administrator-create an EUL.

It will then ask who is to own the EUL, and when you click on the Finish button it is created. You must then wait for a few moments while the metadata is loaded into the database.

There are many steps involved in setting up the environment for Discoverer. To help you remember them, a window will appear, as shown in Figure 6.29, that lists all of these tasks. By clicking on any item in this list, you can automatically start that component.

Figure 6.29: Discoverer Administrator-task list.

Business area

Once the definition of the end-user layer is complete, a business area is named, where all the information needed to query the warehouse must be defined. The purpose of the business area is to group information into business-oriented categories, such as sales or finance, that are familiar to end users. Business areas are the unit of access control and a user can be assigned to one or more business areas. Users will then have access to all of the objects within the assigned business area.

Within this business area, you will specify exactly which data a user may access and how those data may be joined, as well as describe data aggregations and new data items based on calculations on existing data items. Creating the business area will take some time, but it will reap significant benefits later.

When creating the business area, we must first specify the schemas from which data is to be made available to the end user, as illustrated in Figure 6.30. We can select any number of users from the list and also request that it only selects items from the list that match the pattern specified in the box. In our example, we have selected only the user or schema EASYDW.

Click To expand
Figure 6.30: Discoverer Administrator-select the schema.

Once we know the schema from which the data will be available, we can then explicitly state from which tables or views we can retrieve data, as shown in Figure 6.31. Using our Easy Shopping example, we have given the users access to four of the five tables in this data warehouse.

Click To expand
Figure 6.31: Discoverer Administrator-select the tables and views.

One extremely useful feature in Discoverer, is the ability of the wizard to automatically create joins based on primary and foreign keys and create hierarchies from the data, as illustrated in Figure 6.32. By allowing the wizard to perform these tasks, there will be less setup work for you to do, and by default these options are already selected.

Click To expand
Figure 6.32: Discoverer Administrator-automatic joins.

There is one final screen before the first stage in creating the business area is complete. In Figure 6.33 we must name the EUL, and optionally add a description to describe what is being held. Discoverer will also generate names for all of the objects, and in Figure 6.33 you can see the options available, such as capitalize or replace underscores with spaces.

Click To expand
Figure 6.33: Administrator-naming the business area.

Clicking on the Finish button, the first stage in creating our business area is now complete, and the screen shown in Figure 6.34 will now appear. This is where the real work begins. We can now set up the rest of the business area using our task list, shown in Figure 6.29, as a reminder of the steps to complete. Although we have shown only the creation of one business area, you could create any number of business areas in your environment, each with their own unique set of data requirements.

Click To expand
Figure 6.34: Discoverer Administrator-setting up the business area.

Restricting the visible columns

By default, when access is given to a table, all the columns in that table are accessible. What is nice about Discoverer is that users can only access data via the business area. If the table and column are not included, the user will never even know that this data existed.

In Figure 6.34, we can see all the tables from the database that we have access to. Note that our table TODAYS_SPECIAL_OFFERS is now called Todays Special Offer as a result of the naming change requested earlier. By default, the end user will have access to every column in those tables. If you click on the table name to expand it, all the columns in that table will appear. To remove any of those columns, simply click on the item using the right mouse button, and a drop-down list will appear. One of the items in that list is delete item. Simply select that option, and the item will be removed from the business area but not from the database.

Folders

Before moving on, there is some important terminology that you must familiarize yourself with. In Discoverer, a table or view is known as a folder, and a column from the table is called an item. A folder can be one of two types: simple, where it is based on a single database table or view, or complex, where it can contain items from other folders and can be nested

An item corresponds to a column in a relational database. A simple item is based on a single column in the database, but an item can also be calculated or derived based on a formula using other items, functions, or operators.

Changing item details

The attributes of any of the items that you have selected may be modified by selecting that item, clicking on the right mouse button, and then selecting Properties. The window illustrated in Figure 6.35 will then appear, and you can then modify whichever properties you like. In this example, we have changed the item's name from Supplier to Main Supplier, which means that our users can be presented with friendly meaningful names rather than computer format names.

Click To expand
Figure 6.35: Discoverer Administrator-change the item details.

When the business area is first created, if the database has primary and foreign keys defined, then Discoverer will automatically create joins between those constraints. However, you can specify your own joins by selecting Insert from the strip menu and then Joins. The window illustrated in Figure 6.36 will appear.

Click To expand
Figure 6.36: Discoverer Administrator-define a join.

A join is defined by selecting, from the drop-down list, which item from the folder is to be joined and the type of join. In our example in Figure 6.36, we have specified a join between the Todays Special Offers table and the Product table using the column Product_id. The advantage of defining these joins now is that when a user writes a query, Discoverer will know how to join the data, so this is one less piece of information that our user has to supply.

Creating new items

Another feature that many DBAs may require is the ability to create new columns or items in the database by calculating their results from other columns.

A calculation creates a new item in the end-user layer. It will not add underlying columns to database tables; it is used to create a new item where there is no underlying database column containing the required data. Calculations can be simple, such as weight*4.54, or they can be complex mathematical or statistical expressions. For example, in Figure 6.37, a new column called Total Cost is created in the Purchases table by adding together the columns purchase price and shipping charges. You can create as many of these types of calculations as you require, and they will appear as an item for that table.

Click To expand
Figure 6.37: Discoverer Administrator-creating a calculated item.

Hierarchies

We have already seen that hierarchies play an important role in our data warehouse. Although in Oracle we can create dimensions, at the time of writing these are not used by Discoverer, and we must create our own dimensions, which are known in Discoverer as hierarchies. Hierarchies are very important in Discoverer, because if an item is in a hierarchy, then users can drill up, changeing the query to show a higher level of details, or drill down to show more detail.

This is how we were able to drill on our report shown in Figure 6.27.

A hierarchy is created by clicking on Insert in the menu at the top and then selecting Hierarchy. First you will be asked about to the type of hierarchy you want to create, an item or a date. When the business area is first created-it is quite likely that Discoverer will automatically create a timebased hierarchy. Therefore, you will probably only have to create non-timebased hierarchies.

Figure 6.38 shows how easy it is to create the hierarchy by selecting the items and then defining the hierarchy relationship. In Figure 6.38, we have created a very simple hierarchy between product_id and category.

Click To expand
Figure 6.38: Discoverer Administrator-defining a hierarchy.

Item classes-list of values

When the end users are actually querying the data, there are times when it may be helpful if they can see a possible list of values. For example, suppose they want to pick out all of the electrical items. If they know which category they are represented in, then this will facilitate rapid query generation.

An item class can describe the hierarchical relationship between items, a list of values, alternative sort keys for items, and the display methods. Therefore, an item class defines all of the attributes for an item. Once an item class has been defined it can then be assigned to other items that share similar properties.

An item class is created by selecting Insert from the menu at the top and then Insert Class from the list or by clicking on the text in the administration task list, shown in Figure 6.29. Several different types of classes of items may be created, but if you select List of Values, all you have to select is the column containing the values and which tables will use it. When it is complete, click on the tab Item Classes, and the window illustrated in Figure 6.39 will appear. In this example a list of values called Categories has been created from our products dimension table. If we expand the entry categories, the data warehouse can be queried, and all the different values will be displayed. In Figure 6.39, we can see some of the different products that are sold. Here we only have a few items, but in the real world, where you may have a number of different values, you can omit this step of displaying the results.

Click To expand
Figure 6.39: Discoverer Administrator-item class.

When an item class is defined, you can also define the sort order for the data. In Figure 6.39, we have chosen to use the conventional alphabetical method, but that is not always suitable and you can specify a logical order such as N,S,E,W rather than E,N,S,W.

Summaries

We have already seen the importance of creating and using materialized views in our data warehouse in Chapter 4. Discoverer allows you to create your own materialized views, which Discoverer calls summaries, from its Summary Wizard, either by:

  • Using query performance statistics

  • Manually creating the summary

  • Registering a previously built summary

To create your summary, click on Insert in the menu at the top, and then select Summary; the Summary Wizard appears, and you will be asked the type of summary that you wish to create. If you select From Items in the end-user layer, you can create your own summary.

Next, the window illustrated in Figure 6.40 appears. There you select the folders and items from within those tables that are to appear in the summary. In our example, we have selected only the Purchases table, but you could select multiple tables. Then we chose four of the data items in that table. For the item purchase price, we have asked that this value be aggregated. You will see that Discoverer will automatically supply a range of functions for you to select when a function may be applied to an item. In Figure 6.40, for the item purchase_price, we have asked that the SUM function be applied to this item. Next, you are asked to specify which groups of items you require.

Click To expand
Figure 6.40: Discoverer Administrator-creating a summary.

As we have seen, it is very important to ensure that our materialized views contain the latest data. When defining your summary in Discoverer, you can specify how often it is to be refreshed. Remember, that as new data is added to the warehouse, the summaries must be maintained to reflect the latest data. In Figure 6.41, we can see that we have stated that this summary should be refreshed every hour.

Click To expand
Figure 6.41: Discoverer Administrator-refreshing the summary.

If you don't know what summaries to create, then Discoverer can recommend them for you using its own Summary Wizard. In Figure 6.42, we can see one of the steps from the wizard, where we can select the summaries we require based on our space requirements. This wizard is very similar to Summary Management's own Summary Advisor Wizard, described in Chapter 4, but the recommendation process used by Discoverer is different from the one used by the server's Summary Wizard.

Click To expand
Figure 6.42: Discoverer Administrator-Summary Wizard.

Creating either Discoverer summaries or materialized views is very important if you want to achieve the fastest query response time. We have seen here how to create summaries directly in Discoverer. You also can create materialized views, as discussed in Chapter 4, and Discoverer will still use them, because a materialized view doesn't have to be created by Discoverer for it to be used by Discoverer.

Security issues

The final setup task is defining who may access the business area you have just created. You can start this component by double-clicking on Grant Business Area Access, shown in Figure 6.29. A window will appear that will allow you to state which users can access a business area or which business areas a user can access. In Figure 6.43, we can see that the only user we will grant access to in our business area is EASYDW. Don't forget that although in this example we have enabled access via the user name, you can also grant access to the business area via the roles that may be given to a user. When you are satisfied that all the relevant access rights have been given, click on the Apply button to complete the changes.

Click To expand
Figure 6.43: Administrator-granting access to the business area.

Figure 6.44 illustrates our complete business area. Expanding just the Purchases table, we can see all the items that are available to our users, the new calculation that we created, called Total Cost, and the functions that we can apply to that column. At the bottom of the window, we can see that joins have been created from the Purchases table to the Product and Time table. Of course, not all the information can be displayed on this one window, so, to see the Hierarchies tab you will have to click it. The same is true for item classes and summary information.

Click To expand
Figure 6.44: Discoverer Administrator-business area.

We have now completed all of the basic setup tasks for using Discoverer. Don't forget to save all of your work, and please remember that the tool is much more comprehensive than we have shown. For example, we haven't shown that Discoverer fully supports the Oracle Analytical Functions to enable sophisticated analysis of the data. Now, we can start using Discoverer Desktop, Viewer, or Discoverer Plus to retrieve data from our data warehouse.

6.3.3 Query using Discoverer Plus or Discoverer Desktop

Once the environment has been set up for querying via Discoverer, you can start either the Desktop edition or the Plus version, which has been designed for the Web. In this section our examples will use the Web version, Discoverer Plus.

Reports in Discoverer are held in a workbook, so the first step is to connect to the database where our workbook resides. In Discoverer Plus, when you launch Plus from your browser using a URL such as

http://easydw.com:7777/discoverer/plus

you will be presented with a list of the databases you can connect to, as shown in Figure 6.45. Here we can see that we only have one database, called EasyDW.

Click To expand
Figure 6.45: Discoverer Plus-list of databases for connection.

These connections will have been defined previously and comprise your user name and database name. Therefore, all you must supply is your password, and you will be asked whether to create or open an existing workbook. In Figure 6.46, we see the initial window, where we specify the workbook and how the results are to be displayed. Discoverer offers a range of display options, such as showing the data in tabular or a crosstab form. Using the crosstab format is ideal when you have multidimensional data to display.

Click To expand
Figure 6.46: Discoverer Plus-using a workbook.

Now it is time to specify exactly what is to be reported in this query. First, you must select the business area that was defined using the Administrator edition, which determines the data that you may see. In our example, we have only the Sales Data business area, but there could be several to choose from.

Now we have to select the items from those folders. This is a simple process, involving moving them from the left window (available) to the right window (selected). In Figure 6.47, we have selected the item category from the product folder and the date from the purchases folder and then requested that the item purchase price should be aggregated.

Click To expand
Figure 6.47: Discoverer Plus-selecting data to be displayed.

When Discoverer identifies an item upon which functions may be applied, all possible options are made available to the user. For example, in Figure 6.47, we see that for the item purchase_price, we can apply the functions MAX and COUNT, and Discoverer can also calculate the minimum, maximum, or an average. We have already chosen to sum purchase price by simply expanding the aggregate options for purchase price in the left window, then selecing the required aggregate, and clicking on the right arrow. This process is repeated for every item required in the report.

Next, we must specify the layout for our report. This is very easily achieved by dragging the columns to where you require them on the report. In Figure 6.48, we have specified our order as category, date, and total price. But we could easily drag the date column and place it after the total purchases for that category. At this stage we can only decided how the data is to be presented; on the next screen we can specify formats and headings.

Click To expand
Figure 6.48: Discoverer Plus-table layout.

For each item that will be displayed on our report we can now specify how those data is to be formatted and select the heading to be used on reports. In Figure 6.49, we have changed the heading for our sum on purchase_price to be Total Purchases. By clicking on the Format Heading button, the font and alignment options can be defined, and clicking on Format Data allows us to specify how the data is actually presented. In this example we have decided that our total will have no decimal places.

Click To expand
Figure 6.49: Discoverer Plus-format headings.

Now we can specify a condition to limit the results of our worksheet to a specific criterion. There are some very extensive options available here; however, in this example we are going to view all of the data. Then we can specify how to sort all of our data, which is achieved by clicking on the Add button and selecting a column. It then appears in the list, and you can also specify the sort order. In our example in Figure 6.50 we are sorting by date first and then category.

Click To expand
Figure 6.50: Discoverer Plus-sorting the data.

At this stage we could also add calculations to appear in our report, such as profit made by subtracting purchase price from cost price. In our report today we will not include any calculations. Another option is the ability to create a percentage point on any item in the report that can be useful to help understand the data.

We have already said that we want to sum the item purchase price, and in Figure 6.51 we can now add a Total to this item as well. As you can see, there are a number of options available, such as whether we want a subtotal, the type of sum to perform, and how the data should be formatted. Once again, we can create as many totals as we need for inclusion in our report.

Click To expand
Figure 6.51: Discoverer Plus-defining totals.

Clicking on the Finish button will display our data. Discoverer will now query our data warehouse directly and show us the results, as illustrated in Figure 6.52. There we can see the total sales by category for a given date.

Click To expand
Figure 6.52: Discoverer Plus-report.
Hint:

You may have to format the cells of the report to see the data if the numbers are large, because, by default, it uses small numbers.

Note that in order to report this data, we did not have to specify how to join the tables we selected, because the join information had already been specified in the business area. This is one of the really nice benefits of using Discoverer, because the end user does not have to know about relational joins. All of this work has been done behind the scenes by the person who created the business area using the Administrator edition. Now all our user has to do is select the data of interest, answer the questions on a few screens, and then click the Finish button to request the information.

Now that we have our report, we can customize it to our own requirements by either clicking on the items or selecting from the menu at the top. To change the format of our numbers, if we click on Sheet and then Format, Figure 6.53 appears, where we can amend the item. See how all the screens we used to define our report are now shown as tabs, so we can easily select them and amend anything there as well.

Click To expand
Figure 6.53: Discoverer Plus-customize the report.

If you are interested in how the report is being executed within the database, selecting Sheet followed by Show SQL, will bring up the SQL Inspector box shown in Figure 6.54. Here you can either view the SQL used to execute the query or, as shown in Figure 6.54, the query's execution plan.

Figure 6.54: Discoverer Plus-SQL Inspector.

Now that we have our report, we may want to look at the data from a different perspective. For example, we could roll up the data to see sales by month or drill down to see which electrical item generated most revenue.

You can change the report by clicking on the arrow, and the screen shown in Figure 6.55 will appear, which shows you how you can drill on this data. Here we have chosen to show the data by month, but we could select any one of the five options shown. Note that our user did not have to tell Discoverer about how it could report on the data. All this information was previously defined during the setup, so once again our end user needs to know little about how the data is stored in order to get the required information.

Figure 6.55: Discoverer Plus-drilling up/down the data.

In Figure 6.56, we now see our report with the information at the month level; because we have a time hierarchy, we can change our report to sum by year, instead of by day, simply by selecting year from the list. Hopefully, now you are beginning to appreciate all of the setup work that we completed using the Administrator edition.

Click To expand
Figure 6.56: Discoverer Plus-report at the month level.

So far we have only viewed our data in a traditional report format, but Discoverer Plus can also represent our data graphically answering a few questions using the Graph Wizard, which is shown in Figure 6.57.

Click To expand
Figure 6.57: Discoverer Plus-Graph Wizard.

There are over a dozen different types of graphs available from Discoverer Plus, and you can totally customize the output by adding your own titles and legends. We have decided to use a pie chart to represent our monthly sales in Figure 6.58, which makes it easy for us to see that electrical items were the most popular when we ran our sale on January 1.

Click To expand
Figure 6.58: Discoverer Plus-graph of monthly sales.

In this sample Discoverer report, we reported all of the data, but you can select a subset by specifying a condition. What is nice in Discoverer Plus is that you can set up a number of conditions and then select the ones you want for this report. In Figure 6.59, we see one of the screens where you can specify these conditions.

Click To expand
Figure 6.59: Discoverer Plus-select specific data.

There is actually another screen where you actually specify the condition, which can be done using quite complex expressions. Here we can see that we have two conditions defined, and we are using the one for items over $100 to see what effect it has on our report. Then, when we view the new report in Figure 6.60, we can see that it was only electrical items that cost more than $100-and even then, they did not account for most of our sales that month.

Click To expand
Figure 6.60: Discoverer Plus-report using conditions.

Our completed report can also be exported into various formats so that it can be published on our Web site; Chapter 8 explains how to do this.

There are many more facilities available from Discoverer, but, hopefully, you now have an appreciation of some of this tool's capabilities and how ideal this tool is for end users who may not be very computer literate.


Chapter 7: Managing the Warehouse

7.1 What has to be managed

Once the warehouse has been created and is populated with data, it is very important to ensure that it is correctly managed. The warehouse must be configured for optimal performance and availability. Disk, memory, and CPU resources must be managed. In this chapter, we will describe some of the management tasks and provide advice on how to execute them.

Once again, we will make use of the GUI tools provided in Oracle Enterprise Manager (OEM) to manage our database. There are alternative methods to this approach, but we are sure you will agree that using OEM makes managing the database considerably easier.

This chapter will first provide an introduction to OEM, discuss various techniques to back up your data warehouse using Recovery Manager (RMAN), the importance of developing a strategy for disaster recovery, reorganizing the warehouse, gathering optimizer statistics, tuning the ware-house, maintaining security, and monitoring space use.


Managing using Oracle Enterprise Manager

7.2 Managing using Oracle Enterprise Manager

Oracle Enterprise Manager can be used to manage any database, located anywhere on the wide area network. Therefore, from this one tool, you can manage your entire system, no matter where the components might be. This is an extremely powerful capability, and you should remember that what we are describing here could be used not only for the data warehouse but also for other databases on your network. In this section, we'll take a look at some of the concepts and steps required to start using OEM.

Enterprise Manager is designed to run in a three-tier architecture, as follows:

  • The console provides the graphical user interface on the client system.

  • The management servers (OMS) provide administrative functions, such as executing jobs and events, and runs in the middle tier.

  • The intelligent agents monitor the database, start and stop it, and gather performance data. The intelligent agents run in the third tier and are located on the same nodes as the Oracle databases.

All information required to manage and administer the environment is stored in the OEM repository, which can be part of any Oracle database, or in an Oracle database that is used exclusively for Enterprise Manager. It is where the management server stores the information it needs to manage the network configuration, the events to monitor, jobs to run, and the administrator accounts.

7.2.1 The console

All work is performed from the Oracle Enterprise Manager Console. The console either can be launched connecting to the management server or standalone, as shown in Figure 7.1. When you want to use jobs and events and the data management and backup wizards, you must connect to the Oracle Management Server. If you only need to perform basic database functions to manage the schema, instance, storage, and security the console can be launched standalone.

Click To expand
Figure 7.1: Launching the Oracle Enterprise Manager Console.

A typical console is shown in Figure 7.2. The console contains menus, toolbars, and drawers. The toolbar is located on the upper left-hand side of the screen, and the tool drawers are located on the left side of the screen. When an object in the navigator tree is selected, the interface is displayed on the right-hand side in the detail pane.

Click To expand
Figure 7.2: Oracle Enterprise Manager Console.

When you start the console for the first time, or whenever anything changes in your environment, the information shown in Figure 7.2 should be refreshed. This is accomplished by selecting Discover Nodes from the Navigator option on the top toolbar. You will be prompted for the nodes to be searched, and, provided the Oracle Agent process is running on those nodes, it will interrogate your system and advise the console of the databases and listeners that are running. Therefore, you don't have to specify anything. In our example here, we can see that it has discovered the default ORCL instance that contains our data warehouse EASYDW on the node shillson-lap.us.oracle.com.

The Console Navigator tree, shown in Figure 7.3, is probably the most important area on the screen, because it is from here that numerous operations are started. By default, there are five subject areas: databases, groups, HTTP servers, listeners, and nodes.

Click To expand
Figure 7.3: The Enterprise Manager Navigator window.

In addition, in Oracle 9i Release 2, the Navigator tree also provides access to database administration, events, report definitions, jobs, and groups. The database folder in the Navigator tree allows you to administer the database instances, schemas, security, and storage.

In Figure 7.3, the Navigator window has been expanded. In the databases section, it lists every database, irrespective of the node on which it resides. To see on which system a database resides, you have to expand the nodes area and then see which databases are on that node. In our example, it is easy, because we only have one node. If we had many nodes running many databases, however, the view would be more complex. This is when the group feature, which is described shortly, becomes very useful.

From the Navigator window, you can perform a number of tasks with respect to databases. By clicking on the database name, it will expand to display a list of features. When you select one of the features, a description appears in the detail pane on the right side of the console. For example, you can start and stop an instance by clicking on Instance. You can create and drop tables, indexes, and views by clicking on Schema, and you can manage security, storage, and other objects in a similar manner.

An Enterprise Manager administrator account is required to log in to the console with a Management Server connection but not to launch the console standalone. Enterprise Manager has two types of administrator accounts: regular administrators and super administrators, who have additional privileges. An Enterprise Manager administrator account is different from the user accounts on the target databases you want to manage.

A Super Administrator is created when Enterprise Manager is installed and configured. Selecting the Manage Administrators option from the configuration toolbar can create additional administrators. In Figure 7.4, the Super Administrator is creating an account with the user name of EASYDW for the DBA of the EASYDW data warehouse. Access is granted to the job and event systems and, finally, to the target databases the new administrator will manage. Note that this is not a Super Administrator account, because we have not selected that option. We can specify the user name and password for every database and node the DBA's require access to.

Figure 7.4: Creating an Enterprise Manager administrator account.

7.2.2 Why create groups?

Earlier, we spoke of the issues surrounding managing multiple databases, and one solution to this problem is to be able to see a graphical representation of your environment. The purpose of a group is to allow you to visually represent an environment. It doesn't have to be the entire environment; it could be just a part of it.

A group is created from the Object menu. You enter a name and select which objects from the available targets make up the group. You can also select a background image. The group name describes the environment, and the image is required so that you can overlay the objects onto a picture.

In Figure 7.5, we can see our group called Easy Shopping Inc., which represents our warehouse. The underlying graphic is a map of the area in the United States where the warehouse is located. Referring to Figure 7.6, we have added the ORCL database and the machine shillson_lap.us.oracle.com. They have been placed on the map to represent where they physically reside; of course, the choice is yours regarding where to place them.

Click To expand
Figure 7.5: Creating a group.
Click To expand
Figure 7.6: Group—Easy Shopping Inc.

You may be saying to yourself: This all looks very pretty, but what is the point apart from graphically showing the system? In the next section, we will describe events, and when events are combined with the group picture, you have a very powerful management environment.

You can create as many groups as you like, and they are displayed in the navigator tree.

7.2.3 Using events

One of the problems for anyone managing a database is knowing when certain events occur. For example, suppose the database instance suddenly shuts down. Wouldn't you like to know immediately that it has happened rather than wait for the calls from your users, who are complaining that the system is no longer available?

Within the console, you can define events that will allow you to monitor your data warehouse and be notified when certain events occur. An event is created by clicking on Event in the toolbar and then clicking on Create Event. At this point, the screen shown in Figure 7.7 will appear.

Click To expand
Figure 7.7: Creating an event.

Referring to Figure 7.7, we have given our event a name, Warehouse Status, and also added some further comment. This particular event is a database event, but we could select others from the drop-down list, such as node, in the section marked Target Type. Next, we select the ORCL database from the list on the right side and, using the arrow buttons, move it over to the left side. The frequency of monitoring can also be changed, if required. Then we must click on the other tabs at the top of the window to complete the event definition.

Clicking on the Tests tab displays the screen in Figure 7.8, where we select the events to monitor. In this example, we have selected DatabaseUp-Down, so that we are notified whenever the database goes up or down.

Click To expand
Figure 7.8: Selecting events to monitor.

This event has no parameters for us to specify, so the next step is to state who should be notified when this event occurs. It is not mandatory to notify anyone; you could define the event and then, provided the database has been dragged over to the group, the status of the event will be registered there.

For example, in Figure 7.6, the database ORCL has a colored flag against it (which, unfortunately, is very difficult to show in a black-and-white book!). The ORCL database is showing a green flag to denote that it is open. If it were closed, it would have a red flag against it.

This is one of the powerful visual features of creating a group, because you can instantly look at the picture and see the state of your system. In Figure 7.9, we have stated that users SYSMAN and EASYDW should be notified whenever this event occurs. Administration staff can be notified of events via paging and/or e-mail. Both of these options can be configured within the console.

Click To expand
Figure 7.9: Advising users of an event.

By selecting Configuration from the toolbar and clicking on the Schedule tab, the screen in Figure 7.10 will appear.

Click To expand
Figure 7.10: Notification schedule for events.

Once again, this is difficult to see in a black-and-white book, but here you specify how you are to be notified for each hour of the day. Since we know that lunch is always taken at 1:00 p.m., during this period and on Saturday, we have stated that we should be paged whenever this event occurs. However, during the working day, the preferred contact method is e-mail.

Here we have just scratched the surface of what is possible with events. But hopefully it will encourage you to investigate these further and implement them as part of your data warehouse management procedures.

7.2.4 Scheduling jobs

Data warehouses require a great deal of maintenance, because data is continually being loaded. You could write scripts to perform these tasks and remember to run them, or you could use the Job Scheduler in OEM to automatically run your jobs at the designated time.

The job facility provides the ability to define jobs that must be run in order to manage your data warehouse. These jobs can be placed in a library and scheduled automatically by the console to run at the specified time.

To create a job, select Job from the toolbar and then Create Job. The screen shown in Figure 7.11 will appear, which is where you describe the details for this job. In this example, we are creating a job that will automatically close the data warehouse at 7:00 p.m. First, we name the job and state upon which database it is to be performed. Although this example features a database task, other options are available from the drop-down list on destination type.

Click To expand
Figure 7.11: Creating a job.

All jobs may be kept in the job library for reuse. Therefore, if you wish to retain the job, now is a good time to click on the button at the bottom of the window to Submit & Add Job to the library. Choosing this option at this time will merely enable that option and change the submit button to Submit & Add. It is not until you click the Submit & Add button that the job will be submitted to the job queue.

To move to the next screen, click on the Tasks tab to select the action that is to be performed. You can see in Figure 7.12 that many tasks are available. In this example, we have selected Shutdown Database.

Click To expand
Figure 7.12: Selecting task for a job.

Once the task has been chosen, click on the Parameters tab to see if any parameters are needed for this task. In Figure 7.13, we can see that for closing the database, it asks us how we want to close the database. We have selected the Immediate mode.

Click To expand
Figure 7.13: Parameters for a job.

Also note here that we have the ability to specify a different user name and password to perform this operation from the default one that we set up earlier. Since closing the database requires a powerful user name, we have changed it to use the system user name for this task.

The definition of the job is almost complete. We must now click on the Schedule tab to state when this job is to be run. Referring to Figure 7.14, we have stated that this job is run on Monday through Friday, starting from July 3 at 7:00 p.m.

Click To expand
Figure 7.14: Scheduling a job.

Finally, select the Access tab to state who, if anyone, should be notified when this job is run. Then click on the Submit button to place this job on the queue to be run.

To see the jobs that are in the queue select Jobs in the Navigator tree. Once the job appears in the queue, you can monitor the job's progress, as shown in Figure 7.15. In the example in Figure 7.15, the status of the job is Scheduled, because the job is pending. If it were actually running, the status would be shown as Active. Once the job has completed, it will disappear from the active window. Click on the History tab to see whether it succeeded or failed.

Click To expand
Figure 7.15: Watching a job run.

Double-clicking on the job will display the screen shown in Figure 7.16, which is where we can see the details of the job. In Figure 7.16, we can see that the job has been submitted and will run next at 7:00 p.m. on July 3.

Click To expand
Figure 7.16: Logging from the job.

To see the results from the job, select the job, and click on the Show Output button. Figure 7.17 shows a load job that failed, because it could not find the control file in the specified location.

Click To expand
Figure 7.17: Output from the job.

The examples in this section have shown database events and jobs. Alternatively, a job could be an operating system task that needs to be performed for managing the warehouse, such as ensuring that files are copied from one system to another prior to loading the warehouse.

The number of jobs we can submit here is quite extensive, and, once they are in the library, we have a comprehensive set of tasks that are safe and that can never be lost.

Clicking on Job on the toolbar and then selecting Job Library can find the job library.

Hint:

Don't forget to back up the database used to hold the Enterprise Manager repository, or you could lose all of the jobs in the library if there was ever a failure of the database.

7.2.5 Launching applications

The console is not limited to monitoring databases and nodes; we can also launch database applications from here. The licensed options will determine the applications that are available. In Figure 7.18, we can see one of the techniques used to launch an application. Here we have selected the database from the Navigator window. By clicking on the right mouse button, the available tools are displayed, such as Shutdown.

Click To expand
Figure 7.18: Starting applications from the console.

In this example, we have clicked on Backup Management, giving us access to the backup and recovery tools. To perform an export or import of the data, the Data Management option must be selected.

An alternative method for starting the applications is to use the applications drawer located on the left-hand side of the screen. For example, to launch SQL*Plus Worksheet click on the [image] icon. Applications can also be started from the toolbar under the Tools section.

Hopefully, you are now beginning to appreciate how to manage your data warehouse using Oracle Enterprise Manager. Next, we will look at how we can use OEM to perform the management tasks in our data warehouse.


Backup and recovery

7.3 Backup and recovery

One of the most important management tasks for any database is taking backups of the data. It may seem obvious, but you will be surprised how many companies jeopardize their business by taking backups infrequently or by not taking care of their backup tapes.

The problems with backing up a data warehouse are slightly different from backing up a typical production system. Since warehouses usually receive large loads of data, typically overnight, backups have to be scheduled along with this work. Plus, we have the added complication of deciding how to back up the database.

Why is this a problem? Well, normally, a database is backed up in its entirety, but that may not be possible with your data warehouse, especially if it is particularly large (multiple terabyte size). Therefore, incremental backups and backups of one or more tablespaces may be more practical. Careful management of the backup tapes is critical, or you could find yourself unable to rebuild the data warehouse.

Prior to Oracle 8i backups were usually taken using the Operating System Backup Utility by copying all the relevant files. Now you can consider using Recovery Manager (RMAN) to back up your database. RMAN will be discussed in this section.

7.3.1 Backup

In this section, we will discuss the various techniques you can use to back up your data warehouse. However, it is important to realize that if your data warehouse is very large, taking a full backup of it may be almost impossible due to time and other constraints. The following types of backups will be discussed:

  • Full backups

  • Incremental backups

  • Tablespace backups

Creating a backup configuration

Selecting the database from the Navigator window, right-clicking on the database, and then selecting Backup Management will start the Oracle 9i RMAN Backup Manager. A list of tools is provided that includes both backup and recovery; however, start by selecting Create Backup Configuration.

A backup configuration is a set of defaults used for backup operations. It determines where the backup is to be stored, what the backup medium will be (tape or disk), and whether you want to store information describing the backup in a recovery catalog. The recovery catalog will be described later in section 7.3.2.

You can create as many backup configurations as are required for your system. The purpose of this step is to specify where the backup is to be placed and how it is to be named. Figure 7.19 illustrates the first step in this wizard where the name of the backup configuration is specified.

Click To expand
Figure 7.19: Creating a backup configuration.

From here you now click on the various tabs to set up the options. It is not necessary to visit all of the screens, but you will have to click on the Channels tab, shown in Figure 7.19. This is where you specify the location and name of the backup file.

Once again, you can create as many channels as you require. Think of the channel as where you specify all of the physical output attributes, such as whether the backup is to go to a tape or a disk.

When it comes to naming the file, various options are available so that unique file names can be created. In Figure 7.20, we have chosen %s, which returns the backup set number and %t, which returns the backup set time stamp.

Click To expand
Figure 7.20: Selecting the channels in a backup configuration.

Another useful parameter is %u, which generates an eight-character name comprising the backup set number and the time the backup was taken. There is no browse capability for the directory name; therefore, ensure that it is spelled correctly.

Hint:

The directory for the backup must end in a \ or /, depending on your hardware platform.

A complete list of all the options can be found in the Oracle 9i Backup and Recovery Manual. If you are backing up frequently, it is important to name your backup files very carefully; otherwise, you could find that you are unable to recognize a backup file.

In this example, the backup will go to disk, but in a data warehouse, it will have to go to tape. To store backups on tape RMAN requires the installation of a media management library, available from a third-party software supplier such as Legato or Veritas. Refer to Oracle's Backup Solutions Partners Program for the current list of certified vendors.

It is extremely important to ensure that the physical labels placed on the tapes match the contents. Too many times we have experienced problems where it is impossible to recreate the database from the backups, because the labels on the tapes do not match the actual tape contents. Sloppy management practices like this could mean that you lose your entire data warehouse.

Whenever you take a full backup of the system, do not discard the previous full backup set, because if there were a problem with the current backup, you would lose your entire data warehouse. Therefore, save as many full backup sets as is possible before you recycle the tapes.

There are several other tabs that can be selected. If the default user for this database cannot connect as SYSDBA, then you must click on the Preferred Credentials tab to specify a different user name and password.

Full backups

The size of your data warehouse will determine how frequently a full backup is taken. Even if your warehouse is very large, a backup should always be taken periodically.

One of the tools provided in Oracle Enterprise Manager is the Backup Manager, which includes a Backup Wizard, where you can define a pre-defined backup strategy and customize your own strategy.

Predefined backup strategy

The first example shown here illustrates how to set up a predefined backup strategy. The first question asked by the wizard, illustrated in Figure 7.21, is how frequently the backup is to be performed and when it is to be run.

Click To expand
Figure 7.21: Backup frequency.

For those readers unfamiliar with creating backup procedures, the wizard advises on when backups should be taken based on the type of applications using this database. You can see that, for a data warehouse, it is suggesting once a week and picks Sunday, which it expects to be a quiet day, suitable for running backups. Of course, you can change the frequency and day.

The next screen presented by the wizard asks at what time the backup will be run. A review of the options is displayed in Figure 7.22. This is where you specify which backup configuration is to be used. The backup configuration was created in the previous section and is used to specify where the backup is to be stored and whether the data is backed up to tape or disk.

Click To expand
Figure 7.22: Backup configuration selection.

Now we have to choose which database will be backed up. In Figure 7.23, we have selected our data warehouse database EASYDW.

Click To expand
Figure 7.23: Selecting the database.

The definition of the backup procedure is almost complete. In Figure 7.24, we can review all of the options that we have selected. To change any of the values, press the Back key to return to the appropriate screen, and make the necessary modifications.

Click To expand
Figure 7.24: Reviewing the backup procedure.

Once the definition of the backup is complete, it will be placed on the console job queue, as illustrated in Figure 7.25, until it is time for it to run.

Click To expand
Figure 7.25: Backup Job on the console job queue.

Custom backup strategy

An alternative to the predefined backup is to create a custom backup strategy.

The first question asked by the wizard is what type of backup you require, as illustrated in Figure 7.26. If Entire Database is selected, you will then have the choice of either backing up the entire database or taking an incremental backup. An incremental backup could be of interest to a data warehouse user, because it allows you to back up only the data that has changed, instead of the entire data warehouse.

Click To expand
Figure 7.26: Type of custom backup.

Whether or not the database is running in archive log mode will determine the types of backup options available. In Figure 7.26, more options are available because archive log mode is enabled. Enabling archive logging means that all changes to the database are copied into the archive log prior to the redo log being reused.

The steps in a custom backup are very similar to those we have seen previously: You select the type of backup, the channel, when it is to be run, and then name it, as shown in Figure 7.27. This is when you have the option of including this job in the job library.

Click To expand
Figure 7.27: Including backup job in library.

The job will now run automatically and is available for future use because it has been stored in the job library.

Incremental backups

We mentioned earlier that a type of backup very useful in the data ware-house is the incremental backup, because it allows us only to back up the parts of the database that have changed. The screens presented for an incremental backup are very similar to those for a full backup, except that the screen in Figure 7.28 will appear.

Click To expand
Figure 7.28: Incremental backup level.

Here you must specify the incremental backup level. It is extremely important to select the correct level; otherwise, you may get a full backup by mistake.

To clarify how the backup level is specified, refer to Figure 7.29. A complete backup is taken on Sunday; this is our Level 0. At close of business on Monday, a Level 2 incremental backup is taken, which backs up all the changes since Sunday. On Tuesday, another Level 2 incremental backup is taken, which will contain all the changes to the database since Monday. On Wednesday, a Level 1 incremental backup is taken. Since 1 is greater than 0 but less than 2, all the changes since Sunday will be placed in this backup file. On Thursday, a Level 2 backup is taken, so the most recent backup with a value less than or equal to 2 is the Level 1 backup from Wednesday; therefore, this backup will contain all the changes since Wednesday.

Click To expand
Figure 7.29: Selecting an incremental backup.

Using this approach may, at first, seem complicated, but when automatic recovery is used, the Recovery Manager will decide for itself which complete and incremental backups are needed to recover the database.

Tablespace backup

Another technique for backing up the database is a tablespace backup. Using this technique, you have the ability to back up a specific tablespace, which, in a data warehouse, could make for a very nice backup strategy.

In Figure 7.30, we see one of the screens from the Backup Wizard asking us which tablespaces we want to back up. In our example, we have selected the two tablespaces that contain the January data.

Click To expand
Figure 7.30: Selecting a tablespace to back up.

Here is a good example of where designing the database for management should be considered during the initial construction process. If we can group our updates to the database by date, and we know that after a given point in time there will be no changes made within this tablespace, then, if we set the tablespace to be read-only and back it up, we can rest assured that we have captured the data in that tablespace. Since it will never change, we won't have to back it up again.

Changing the status of a tablespace to read-only is very simple. You can either enter the SQL command, shown in the SQL Text box in Figure 7.31, or select Storage on the Navigator tree, select the tablespace, and then click on the box at the bottom of the screen marked Read Only. Finally, clicking on the Apply button will make this a read-only tablespace. You can change it back to a read/write tablespace at any time.

Click To expand
Figure 7.31: Making a tablespace read-only.

Of course, it makes sense to periodically take full backups, just in case there are any problems with the backup files you have taken. Taking only tablespace backups is okay, but it still makes it possible for somebody to accidentally overwrite the tape containing all of the data for a given month.

Backup file sizes

When a database is being built, designers are constantly aware of the size of the database. However, that information seems to get lost when people start thinking about backing up the database. Everyone tends to say: Well it's 300 GB, so it will take x minutes to back up. What people forget to add is that the backup file will need y gigabytes of space.

In the examples shown here, we have used disks to store our backups. For a real data warehouse, however, you will be storing them on tape; so don't forget how many tapes you will require for the backup strategy you will be implementing. It has been known for sites to actually run out of tapes! Our Easy Shopping database is small and occupies approximately 1.33 GB on an NT system. In Figure 7.32, we can see the different sizes for the various backup files.

Click To expand
Figure 7.32: Backup file sizes.

A full backup occupies 527 MB, which is about 25 percent of the full size of the database. Therefore, you can see that you can save a significant amount of storage space by using the backup provided by Oracle 9i. If a backup were taken using the Operating System backup utility, which may or may not compress the file, then 1.33 GB of storage would be needed, instead of 527 MB.

The even smaller files are the incremental and tablespace backups, and here we can see the big advantage of using this utility.

Admittedly, only a few changes were made to the database. But you can see that even for the incremental backups, the size of the file ranged from 3.85 to 10.8 MB, which is a big difference from the requirements for a full backup.

By taking incremental backups, you can make backups of your ware-house more frequently and not have to worry about when you are going to find the time to back up the entire warehouse.

One final reminder on this topic: During the testing of the data ware-house, don't forget to test your backup procedures and obtain some estimates of possible run times for backups. Then you can discuss your requirements with the operations department, so that your management tasks can be included with all of the other work that has to be done.

7.3.2 Creating a Recovery Manager catalog

Information describing your RMAN backups can be stored in the target database's control file or in a recovery catalog. The recovery catalog should always be placed in a database different from the data warehouse. It can be stored in the same database as your OEM repository. Using a recovery catalog is a safer alternative than storing it in the control file alone, since the information is stored separately from the database. It also allows you to store information about all of your backups in a central place.

When creating a backup configuration using OEM, there is a tab to specify whether a recovery catalog is to be used. In the example shown in Figure 7.33, we have stated that the recovery catalog is to be used. It can be found in the database orcl.us.oracle.com, and we must supply a user name and password to access that information.

Click To expand
Figure 7.33: Recovery catalog.

As mentioned previously, the recovery catalog should always be placed in a different database from the one being backed up. We have used the same database for illustration purposes only. Also, don't forget that the recovery catalog must also be backed up regularly, because it is stored inside a database.

Creating a recovery catalog

A recovery catalog should be stored in its own tablespace. In our example, the OEM_REPOSITORY tablespace is used. A new user, who will own the recovery catalog schema in the recovery catalog database, should be created. In the following example we have created a user called rman_user and granted it the appropriate privileges.

Create the user

SQL> CREATE USER rman_user IDENTIFIED BY rman_user
     TEMPORARY TABLESPACE temp
     DEFAULT TABLESPACE oem_repository;

SQL> GRANT recovery_catalog_owner TO rman_user;
SQL> GRANT connect,resource TO rman_user;

The next step is to actually create the recovery catalog, which is achieved by running the RMAN utility. As soon as the RMAN prompt appears, as illustrated in the following code, connect to the database that will hold the catalog. Since you are using the syntax CONNECT CATALOG, an error message will appear saying that the recovery catalog is not installed. Ignore this message. Then issue the CREATE CATALOG command. Once complete, you are now ready to use the catalog.

rman

Recovery Manager: Release 9.2.0.1.0 - Production

Copyright <c> 1995, 2002, Oracle Corporation. All rights reserved.

RMAN> CONNECT CATALOG rman_user/rman_user;
Connected to recovery catalog database
Recovery catalog is not installed

RMAN> CREATE CATALOG TABLESPACE oem_repository;
Recovery catalog created

Before a database can be included in the recovery catalog, it must be registered. Otherwise, you may see the message, Rman-20001 target database not found in recovery catalog. Do not be alarmed by the message.

To register the target database, you need to connect to both the database where the recovery catalog is held and the target database, which is the database to be backed up, in the same session, as shown in the following example.

RMAN> CONNECT TARGET easydw/easdydw@orcl;
Connected to target database: ORCL <DBID=9915458574>
RMAN> CONNECT CATALOG rman_user/rman_user@orcl;
Connected to recovery catalog database

When you register the target database, RMAN populates the catalog tables with information from the control file. The database ID, database name, tablespaces, data files, redo logs, and archive logs are examples of information stored in the catalog. The output shows a full resynch, indicating the catalog tables are updated with the latest information in the control file.

RMAN> REGISTER DATABASE;
Database registered in recovery catalog
Starting full resynch of recovery catalog
Full resynch complete

Once the database has been registered in the recovery catalog, the catalog can then be used to report information about the backups that have been taken.

To see what databases have been registered in the recovery catalog use the LIST INCARNATION command. In the following example, one database has been registered. If there had been multiple databases registered in the catalog, they would be listed here.

RMAN> LIST INCARNATION;
List incarnation;

List of Database Incarnations
DB Key  Inc Key  DB Name  DB ID        CUR  Reset SCN  Reset Time
------- -------  -------- ------------ ---  ---------- ----------
1       2        ORCL     9915458574   YES  190578     05-JUN-02

Once inside the RMAN utility, you can do many things in addition to backing up the database. To obtain information about the tablespaces in the database to be backed up, use the REPORT SCHEMA command, as shown in Figure 7.34.

Click To expand
Figure 7.34: RMAN schema report showing the target database.

The LIST command can be used to report various backup information. The example in Figure 7.35 shows the LIST BACKUP command, which shows the datafiles backed up as part of a full backup taken on July 7, and stored in the file located at C:\DW_BACKUP\EASYDW_466605312_2.

Click To expand
Figure 7.35: RMAN lists the contents of a backup.

Another useful command is REPORT. You can see if there are any orphan backups, obsolete backups, and backups that need more than a certain number of days of archive logs. Obsolete backups are no longer needed, either because they are redundant or because they are not needed for recovery within a recovery window. Orphan backups are no longer usable to restore the database, because they belong to a prior incarnation of the database. Figure 7.36 shows a report of files whose recovery needs more than two days of archived logs. Once again, there are many options available with the REPORT command, and it is suggested that you consult the Oracle 9i Backup and Recovery Manual for detailed information.

Click To expand
Figure 7.36: RMAN report files needing backup.

Although this has been a brief introduction to the recovery catalog, hopefully you can begin to see some of the benefits of using it, especially being able to see which backups exist. Using this information, you will see later how automatic recovery is possible, although, for our data warehouse, we may prefer to do it manually.

7.3.3 Restore

Restoring a database from a backup is a task that most DBAs probably fear. They are always concerned that the backup may fail, leaving them with no database. Unfortunately, unless you test every backup, you can never be sure that a backup file will actually work. If you can, it is a good idea to periodically restore your backup files onto a test system so that you know that the files and your procedures are good.

Hint:

If you are restoring during a serious database problem, try to restore to another location so that if the restore operation fails, you still have the original database.

One of the first problems you encounter when restoring a database is knowing what backup files to use. If you have been using the recovery catalog, then it is very simple, since the recovery catalog will automatically figure it out; otherwise, you will have to check your records to determine the correct backup file.

Don't forget that restoring a database could require a number of full and incremental backup files. This is when the RMAN Recovery Wizard is extremely useful, especially if you want to recover to the latest point in time.

The Recovery Wizard can be started in a number of ways:

  • By selecting it from Tools on the toolbar

  • By using the right mouse button when selecting database from the Navigator window

  • By checking on the database tool icon

Irrespective of the method chosen, the Recovery Wizard will start, the screen in Figure 7.37 will appear, and you will be asked for the type of restoration that is required.

Click To expand
Figure 7.37: Selecting type of restore.

In Figure 7.37, we have the option of restoring the entire database, or restoring specific tablespaces, data files, or archive logs. Recovery can be to a specific point in time, such as to Monday at 11:30 a.m. Usually this type of recovery is unnecessary. If a specific problem corrupted the database, however, then you may want to restore to just prior to the job running against the database. In this example we have chosen to restore a tablespace.

After selecting which tablespaces should be restored, it will ask where the files are to be restored to, as illustrated in Figure 7.38. Depending on the reason for the restore, you may want them to go to another location, which can be specified here. The other advantage of this screen is that it provides an opportunity to check which files are going to be restored.

Click To expand
Figure 7.38: Location of data files.

The next step illustrates the reason why you should be familiar with your restoration procedure, because the recovery configuration screen will appear, as shown in Figure 7.39. The wizard automatically selects a configuration. Since backup files are the ones that exist, full backup is selected.

Click To expand
Figure 7.39: Recovery configuration.

Using this configuration, the wizard knows where the backup files are located and is given the information to access the recovery catalog.

You will now have to respond to other screens to check what is to be restored. Finally, a job to perform the restore will be submitted to the Enterprise Manager job queue. You can now monitor the job from the console.

When the job is finished, you can check the restore by viewing the log from the job, which is available from the Job sections of the console under Job History. If you double-click on the job of interest, the various stages of the job are displayed. If you then double-click on the line that says completed, the log of the job will be displayed, as shown in Figure 7.40.

Click To expand
Figure 7.40: Recovery log.

In this example we can see only a portion of the log, but note that because we were using the recovery catalog, the utility automatically worked out which backup files were required. Although we have illustrated how to recover a tablespace, the procedure is almost the same for a data file or the entire database.

Hopefully, this section has given you some idea of how to back up and recover an Oracle 9i data warehouse. This is such an extensive subject that it is highly recommended that you read the Oracle 9i Release 2 Backup and Recovery Manual for detailed information on functionality and many more ideas on how to design, create, and run backup and recovery operations.


Disaster recovery

7.4 Disaster recovery

As data warehouses become more operational in nature and need to be available 24 hours a day, even temporary outages have the potential to jeopardize the business. Just as with an OLTP system, it is critical to develop a strategy to minimize downtime of the data warehouse and develop a plan to recover from disasters that result in the loss of data. September 11, 2001, highlighted the criticality of having a business continuity plan in place.

It is important to consider what may go wrong, examine the expected cost of downtime and data loss, and compare that with the cost of the solution. The investment needed to enable business continuity in the face of a disaster is analogous to buying an insurance policy. Hopefully, you never need it, but it is there if and when you do. In many cases the resources invested to improve availability can also be used for other purposes.

When we talk about disaster recovery, we are referring to a catastrophic failure, such as a building catching fire or a bomb destroying the building. Other problems such as a hardware failure caused by a disk crash should be recoverable from your standard backup and recovery procedures. Disaster recovery is different, because it involves setting up your computer systems at an entirely different site, which presents a whole new set of problems.

The disaster recovery plan for your data warehouse is probably different from your transaction processing systems. One of the main issues with the data warehouse is its sheer size. Hopefully, at least one recent set of backup tapes is kept off site and those kept on site are held in a fireproof safe. Once you know that you have a good set of backup tapes to restore the database, your procedures should take into account the size of the database and the hardware required to rebuild it.

The importance of your data warehouse to the business may determine whether it is even recreated at your new temporary location. It may be that you decide that the business can do without the warehouse until your regular site is available again. Great care should be taken if you make this decision, because it may take much longer than you think to return to normal working conditions. Chances are that if you have a major failure, especially something that relates to a building, it could be many months before you return.

Once you have decided upon your disaster recovery plan, Oracle Data Guard can be used to help implement the plan.

7.4.1 Oracle Data Guard

Oracle Data Guard creates and maintains a copy of a database as a standby at another site. If the primary database becomes unavailable, applications can be restarted and run on the standby. The standby is initially created from a backup of the primary database. As changes are made on the primary database, the redo that is generated is transmitted to the standby, where it is applied, keeping the databases synchronized. Data Guard can be configured to ensure that no data is lost in the event of a failure. It can also be used to support planned maintenance operations such as hardware or operating system upgrades. Both the primary and standby systems can use real application clusters. There are two types of standby databases: physical and logical.

Physical standby databases have been available for a number of releases and are currently used to protect many OLTP systems and operational data stores. Because a physical standby database cannot be open at the same time it is being maintained, the database cannot be used for other purposes. Logical standby databases were introduced in Oracle 9i Release 2 to help solve this problem.

In Oracle 9i Release 2 a standby database may either be a physical standby database or a logical standby database, as follows:

  • A physical standby database is an identical copy, block for block, of the primary database. It is kept synchronized with the primary database by recovering the redo data from the primary database (called redo apply) using the same process that is used for media recovery. When the standby is not running recovery, it can be open for read-only queries.

  • A logical standby database contains the same logical data as the primary database, but its physical structure may be different. It is kept synchronized with the primary database by converting the redo data to SQL and then executing the SQL statements (called SQL apply). Data can be read at the same time SQL apply is applying changes; however, data cannot be updated. On a logical standby database you can also add indexes and materialized views to improve query performance. Additional tables may also be added. These tables can be updated at any time. At the time of writing, there are a number of restrictions on the data types, types of tables, and operations supported on a logical standby.

A Data Guard configuration can be made up of one primary database and up to nine logical or physical standby databases. A typical configuration is shown in Figure 7.41. The primary database transmits redo to both a physical and logical standby database, which may be at the same or different locations. The physical standby is used for disaster recovery, and the logical is used primarily for reporting but can also be used for disaster recovery.

Click To expand
Figure 7.41: Data guard configuration.

7.4.2 The data warehouse and Enterprise disaster recovery plan

The data warehouse could be incorporated into the Enterprise disaster recovery plan in a variety of ways:

  • The logical standby database could be used as a source of data for the warehouse. Since it contains a copy of the data from the production system, it could be used as the source for the ETL process.

  • The logical standby database could be used as the operational data store. Data could then be moved into a staging area on a different system, where it is transformed into a star schema and is ready for ware-house queries.

  • The logical standby database could contain the warehouse tables. The tables maintained by SQL apply must have the same logical structure as the tables on the OLTP system, but additional tables could be created in the logical standby database to perform transformations and create a star schema. Any tables not being maintained by SQL apply can be updated.

The importance of your warehouse to the business will ultimately determine the strategy put in place for disaster recovery. We have now caught a glimpse of how even a data warehouse should be included in our disaster recovery plans and the features provided by Oracle that allow us to implement that plan.


Reorganizing the warehouse

7.5 Reorganizing the warehouse

Reorganizing a database, irrespective of whether it is a data warehouse or a database used for transaction processing-style systems, is not a task to be undertaken lightly. -Unfortunately, in a data warehouse, the time required to reorganize can become a serious issue due to the high data volumes involved. Therefore, it shouldn't be necessary to reorganize the database entirely, but minor changes may be necessary.

7.5.1 Why reorganize?

Reorganizations can occur for a variety of reasons, such as:

  • The business needs change

  • A flaw in the database design

  • A regular archiving of data

  • The government changes the rules

  • The integration of another company's computer system following an acquisition

A change in the business requirements from the system is almost impossible to predict, and one solution to the problem may be to create a data mart rather than restructure the entire data warehouse.

A fundamental flaw in the database design can be overcome by carefully reviewing the database design and, if necessary, asking for an external review by an experienced data warehouse designer. One of the problems with data warehouses is that what may seem a good design at the outset may prove unsuitable in the long term due to the large volumes of data involved. Therefore, try to minimize the likelihood of this occurring by using techniques such as partitioning.

Some reorganization may be planned. If you want to keep your data warehouse somewhat constant in size and not let it grow indefinitely, you may decide to keep only a few years of data. When new data is added, old data is archived and removed. This is called a rolling window operation and is discussed later in this section.

7.5.2 Partition maintenance

Some of the structural changes required by the data warehouse can be achieved using partition operations. In this section, we will look at some of the most common partition maintenance operations, including the following:

  • Adding and dropping partitions

  • Exchanging a partition

  • Splitting and merging partitions

  • Coalescing a partition

  • Truncating a partition

  • Moving a partition

Adding and dropping partitions to perform rolling window operations

When you first create the warehouse, you may have little or no historic data to populate it. Therefore, for the first 18 to 24 months, you happily add new data every month until the required system limits are met. Then, when the data warehouse is full, every month you back up the old data and drop it to make room for new data. Without partitions you would have to scan the data and manually delete it, which is very time consuming. The faster alternative to this problem is to drop the partition containing the old data and create a new partition for the new data, as shown in Figure 7.42.

Click To expand
Figure 7.42: Partition maintenance.

This technique is applicable only if the data is partitioned on a date. Therefore, if you decide upon another partitioning scheme, such as a code, then this type of maintenance operation would not be possible.

The SQL commands to perform the tasks in Figure 7.42 are shown below. First, the old partition containing the data for sales for the month of May 1997 is dropped.

SQL> ALTER TABLE easydw.purchases
     DROP PARTITION purchases_may97;

Next, the new partition for the data for sales for the month of April 2002 is created. The first step is to create the tablespace where the data will reside.

SQL> CREATE TABLESPACE purchases_apr2002
     datafile 'E:\EASYDW\PURCHASESapr2002.f'
     size 5m reuse autoextend on default storage
     (initial 16k next 16k pctincrease 0 maxextents unlimited);

The next step is to alter the table definition to reference the new tablespace that was just created. The PURCHASES_APR2002 partition is added to the table.

SQL> ALTER TABLE easydw.purchases
     ADD PARTITION purchases_apr2002
       values less than (TO_DATE('30-04-2002', 'DD-MM-YYYY'))
       pctfree 0 pctused 99
       storage (initial 64k next 16k pctincrease 0)
       tablespace purchases_apr2002 ;

Exchanging a partition

After a new partition is added to a table, data can be loaded into it. Various techniques to load data were described in Chapter 5. If the data is already in a table in the database, the fastest way to move the data into the new partition is using exchange partition, which is used to move data from a nonpartitioned table into a partitioned table. Exchange partition can also be used to convert a partition into a nonpartitioned table or between various types of partitioned tables. The following code segment shows moving the data from the APR_ORDERS table into the PURCHASES_APR2002 partition of the EASYDW.PURCHASES table.

SQL> ALTER TABLE easydw.purchases
     EXCHANGE PARTITION purchases_apr2002 WITH TABLE apr_orders;

Merging partitions

You can merge two partitions into one or split a partition if a partition becomes too large. Data can either be merged into a new partition or into an existing partition.

One technique often used by designers is to keep the first six months' data in monthly partitions, and then after that, store the data in quarterly partitions. Using the MERGE PARTITION option the data can be easily moved to the new partition, as illustrated in the following code segment.

If we wanted to combine the data for January through March 2002 into a partition for the quarter, we could merge the partitions. First, a new tablespace is created to store the Q1 purchases.

SQL> CREATE TABLESPACE purchases_q1_2002
       DATAFILE 'c:\ora9ir2\oradata\orcl\PURCHASESQ12002.f' SIZE 5M
       REUSE AUTOEXTEND ON
       DEFAULT STORAGE
        (INITIAL 64K NEXT 64K PCTINCREASE 0 MAXEXTENTS UNLIMITED);

The partitions are merged two at a time. If a table is partitioned by range, only adjacent partitions can be merged. We can merge January and February or February and March but cannot merge January and March.

The new partition inherits the upper bound of the two merged partitions. Therefore, the two partitions with the highest ranges need to be merged first (February and March) into the Q1 partition. The following statement merges the partitions and stores them in the newly created tablespace. The PURCHASES_Q12002 partition is automatically added to the purchases table. The PURCHASES_FEB2002 and PURCHASES_MAR2002 are automatically dropped from the purchases table.

SQL> ALTER TABLE purchases
       MERGE PARTITIONS purchases_feb2002,purchases_mar2002
       INTO PARTITION purchases_q12002
       TABLESPACE purchases_q1_2002

Next, the partition with the lowest range, PURCHASES_JAN2002, is merged into the PURCHASES_Q12002 partition. This statement shows merging data into an existing partition.

SQL>ALTER TABLE purchases
      MERGE PARTITIONS purchases_jan2002,purchases_q12002
      INTO PARTITION purchases_q12002
      TABLESPACE purchases_q1_2002

After this operation, the purchases table contains one partition, as shown in Figure 7.43. The high value for the purchases_q12002 table is March 31, 2002.

Click To expand
Figure 7.43: Merge partition.

When merging partitions, both the data and the indexes are merged. The index partitions for January, February, and March were automatically dropped and were replaced by a new index partition for Q1. In the following query, there is a new index partition, PURCHASES_Q12002, for the PURCHASE_PRODUCT_INDEX and the PURCHASE_TIME_INDEX.

SQL> SELECT INDEX_NAME, PARTITION_NAME, STATUS
FROM USER_IND_PARTITIONS;

INDEX_NAME              PARTITION_NAME      STATUS
----------------------- ------------------- -------
PURCHASE_PRODUCT_INDEX  PURCHASES_Q12002    UNUSABLE
PURCHASE_TIME_INDEX     PURCHASES_Q12002    UNUSABLE

The new index partitions are unusable and must be rebuilt, as shown in the following example.

SQL> ALTER INDEX purchase_product_index
REBUILD PARTITION purchases_q12002
SQL> ALTER INDEX purchase_time_index
REBUILD PARTITION purchases_q12002

Splitting partitions

If a partition becomes too big, it may need to be split to help maintenance operations complete in a shorter period of time or to spread the I/O across more devices. A partition can be split into two new partitions. If the PURCHASES table was originally partitioned by quarter, and sales significantly exceeded expectations, resulting in a very large partition, the partition could be split up into three monthly partitions. Tablespaces are first created for PURCHASES_JAN2002, PURCHASES_FEB2002, and PURCHASES_MAR2002.

In the following example, all rows with PURCHASE_DATE less than or equal to January 31, 2002, will be split into the PURCHASES_JAN2002 partition. The remaining rows will remain in the PURCHASES_Q12002 partition. The January purchases will be stored in the PURCHASES_JAN2002 tablespace.

SQL> ALTER TABLE purchases
     SPLIT PARTITION purchases_q12002
     AT (TO_DATE('31-JAN-2002','dd-mon-yyyy'))
     INTO (PARTITION purchases_jan2002
           TABLESPACE purchases_jan2002,
           PARTITION purchases_q12002)

After the split operation, there are two partitions. Next, the remaining rows in the partition, PURCHASE_Q12002 are split into the February and March partitions.

SQL> ALTER TABLE purchases
     SPLIT PARTITION purchases_q12002
     AT (TO_DATE('28-FEB-2002','dd-mon-yyyy'))
     INTO (PARTITION purchases_feb2002
           TABLESPACE purchases_feb2002,
           PARTITION purchases_mar2002
           TABLESPACE purchases_mar2002)

The data has been repartitioned, and the PURCHASES_Q12002 partition automatically dropped, as can be seen in Figure 7.44.

Click To expand
Figure 7.44: Split partition.

Any partitions of the local indexes corresponding to the PURCHASES_Q12002 partition have been dropped. In their place are new local index partitions for the new table partitions.

SQL> SELECT INDEX_NAME, PARTITION_NAME, STATUS
     FROM USER_IND_PARTITIONS;

INDEX_NAME               PARTITION_NAME      STATUS
------------------------ ------------------- ------
PURCHASE_PRODUCT_INDEX   PURCHASES_JAN2002   USABLE
PURCHASE_PRODUCT_INDEX   PURCHASES_FEB2002   UNUSABLE
PURCHASE_PRODUCT_INDEX   PURCHASES_MAR2002   USABLE
PURCHASE_TIME_INDEX      PURCHASES_JAN2002   USABLE
PURCHASE_TIME_INDEX      PURCHASES_FEB2002   UNUSABLE
PURCHASE_TIME_INDEX      PURCHASES_MAR2002   USABLE

Any unusable indexes must be rebuilt.

SQL> ALTER INDEX PURCHASE_PRODUCT_INDEX
     REBUILD PARTITION purchases_feb2002
SQL> ALTER INDEX purchase_time_index
     REBUILD PARTITION purchases_feb2002

Coalescing hash partitions

Range and list partitions can be merged, but hash partitions cannot; they must be coalesced. Rather than determining what partition a row is stored in by comparing the value of the partitioning key with the table's partitioning criteria, as is done for range or list partitioning, the partition is determined by applying a hash function.

Merging partitions, in effect, reduces the number of partitions by one. When the number of partitions changes, the hash function must be reapplied to redistribute the data. Coalescing the partitions does this.

The following example shows the creation of a hash-partitioned table with ten partitions.

CREATE TABLE easydw.hash_purchases
 (product_id            varchar2(8),
  time_key              date,
  customer_id           varchar2(10),
  purchase_date         date,
  purchase_time         number(4,0),
  purchase_price        number(6,2),
  shipping_charge       number(5,2),
  today_special_offer   varchar2(1))
PARTITION BY HASH(product_id)
PARTITIONS 10;

To reduce the number of partitions by one, issue the following command.

SQL> ALTER TABLE easydw.hash_purchases COALESCE PARTITION;

This, in effect, drops a partition.

Likewise, a hash-partitioned table cannot be split when it becomes too big. To increase the number of partitions in a hash-partitioned table, alter the table and add a partition to it.

Truncating partitions

Sometimes we need to remove all the rows in a partition. For example, we may only keep 18 months' worth of data, and, once a month, we need to remove that old data. Rather than delete each row individually, we can use the TRUNCATE PARTITION option, which rapidly removes the data.

To remove all the rows from a partition, but not the partition itself, use TRUNCATE partition. This is much faster than deleting each row in the partition individually. Any local indexes for the partition, such as the EASYDW.PURCHASE_TIME_INDEX, are truncated also. Prior to truncating the partition, there are 7,498 rows in the PURCHASES_JAN2002 partition.

SQL> SELECT COUNT(*) FROM purchases
     WHERE time_key BETWEEN TO_DATE('01-JAN-2002') AND
                            TO_DATE('31-JAN-2002')

  COUNT(*)
  --------
      7498

SQL> ALTER TABLE PURCHASES
     TRUNCATE PARTITION purchases_jan2002;

After truncating the partition, all rows have been deleted.

SQL> SELECT COUNT(*) FROM purchases
     WHERE time_key BETWEEN TO_DATE('01-JAN-2002') AND
                            TO_DATE('31-JAN-2002')

  COUNT(*)
  ----------
           0

Moving partitions

A partition can be moved from one tablespace to another. If the January purchases partition had been created in EASYDW_DEFAULT or some other tablespace, and had to be moved to the PURCHASES_JAN2002 tablespace, the following command could be used. By specifying NOLOGGING, the operation will not be logged, resulting in better performance. After performing NOLOGGING operations, don't forget to take a backup, since you will not be able to recover in the event of a media failure.

SQL> ALTER TABLE purchases MOVE PARTITION purchases_jan2002
     TABLESPACE purchases_jan2002 NOLOGGING

Partitions facilitate management

Most of the management operations that can be performed at a table level can also be performed on an individual partition or subpartition of a table. Partitions or subpartitions can be backed up, exported, restored, and recovered without affecting the availability of the other partitions or subpartitions. To best manage each partition or subpartition independently, they should each be stored in their own tablespace. Each tablespace should be stored on one or more separate storage devices.

Summary Management and the query optimizer make use of the fact that the data is partitioned in choosing the optimal strategy for executing a query or refreshing a materialized view. Partition change tracking (PCT) for materialized views, which was discussed in Chapter 4, keeps track of what partitions have been updated after a partition maintenance operation and recomputes only that portion of the materialized view when it is refreshed. Partition change tracking also increases the query rewrite capabilities of the materialized view by rewriting queries to use the partitions of the materialized view that are not stale. If a table is partitioned, the query optimizer can use a technique called partition elimination, or partition pruning, to determine if a certain query can be answered by reading only specific partitions.

Hint:

The examples shown in this section have performed partition maintenance operations on a table, but don't forget that partitions are also used on indexes, so you will have to create and maintain the corresponding index partitions. Also, in these examples we've used partitions, but subpartitions can be used as well.

7.5.3 Index changes

Probably one of the aspects of the design that will be changed is the indexes. New ones will be created, and existing ones will be modified. In a data warehouse, there is a temptation to create more indexes, because of the lack of updates to the system. However, you should consider the impact all of these indexes will have on the data load time.

If you need to rebuild an index for any reason, then it is suggested that you use the ALTER INDEX REBUILD statement, which should offer better performance than dropping and recreating the index.

Many table maintenance operations on partitioned tables invalidate global indexes, and they are marked UNUSABLE. You must then rebuild the entire global index or, if partitioned, all of its partitions. To avoid this in Oracle 9i Release 2, you can include UPDATE GLOBAL INDEXES in the ALTER TABLE statement for the maintenance operation. Specifying this clause tells Oracle to update the global index at the time it executes the maintenance operation DDL statement.

Hint:

By partitioning the data, you can perform maintenance on specific index partitions rather than on the entire index.


Refreshing the warehouse

7.6 Refreshing the warehouse

An extremely important management task is refreshing the warehouse with the latest data. This is a management task that is usually performed overnight, with the data presented in batches. As was described earlier, frequently the data has to be cleansed and transformed before it can be loaded. Chapter 5 described various techniques that can be used for loading the new data into the warehouse.

Throughout this book, we have seen extensive use of Oracle Enterprise Manager for managing and controlling many of our management tasks. Depending on how complex your tasks are and on the data dependencies, you may prefer to use your own techniques. Otherwise, you could place the jobs on the console job queue and come in the next day to see that everything has run smoothly. For example, it is not uncommon to receive data loads from various sources at different times. If there are dependencies between the data, then you may have to control that within your own management suites.

Once the data has been loaded into the warehouse, the materialized views must be refreshed. Depending on the number of materialized views, this could also take a significant amount of time.

An OEM job could be created to refresh the materialized views. Executing the dbms_mview.refresh_all_mviews procedure could refresh all of the materialized views. Alternatively, you could use the dbms_mview.refresh_dependent option to request that it refresh only materialized views dependent on certain tables. Or you can call the refresh procedure and specify which materialized views are to be refreshed. Specific details on how to refresh materialized views were described in Chapter 4.


Gathering optimizer statistics

7.7 Gathering optimizer statistics

Missing or stale optimizer statistics are often the cause of suboptimal query performance. The cost-based optimizer uses statistics such as the cardinality of the table, number of distinct values of a column, and the data distribution to determine the cost of an access path. The cost is a measure of how much I/O, CPU time, and memory will be required to execute the query. To use the cost-based optimizer effectively, statistics describing the cardinality and data distribution must be collected for each table, index, and materialized view.

Statistics should be collected using the DBMS_STATS package. (Note that previous releases of Oracle had a command called ANALYZE, which is now deprecated.) Statistics either may be gathered by reading all rows or estimated by reading only a sample of rows or blocks. The DBMS_STATS package provides procedures to collect statistics at a database, schema, or table level and also at the level of a table partition.

To collect statistics on all tables and indexes in the EASYDW schema issue the following query:

SQL> EXECUTE DBMS_STATS.GATHER_SCHEMA_STATS ('EASYDW');

If you are just interested in a specific table-PURCHASES, for example-you can use the following statement. This will gather statistics on the table, its columns, and indexes.

SQL> EXECUTE DBMS_STATS.GATHER_TABLE_STATS ('EASYDW',
                                            'PURCHASES');

You can see the table and column statistics by querying the user_tables and user_tab_cols dictionary views, as follows:

-- table statistics
SQL> SELECT NUM_ROWS, BLOCKS, AVG_ROW_LEN, LAST_ANALYZED
FROM USER_TABLES WHERE TABLE_NAME = 'PURCHASES';

  NUM_ROWS     BLOCKS AVG_ROW_LEN LAST_ANAL
---------- ---------- ----------- ---------
     94113        623          46 07-AUG-02

-- column statistics
SQL> SELECT NUM_DISTINCT, NUM_NULLS, AVG_COL_LEN
FROM USER_TAB_COLS WHERE TABLE_NAME = 'PURCHASES';

COLUMN_NAME                    NUM_DISTINCT  NUM_NULLS AVG_COL_LEN
------------------------------ ------------ ---------- -----------
PRODUCT_ID                              162          0           7
TIME_KEY                                 20          0           8
CUSTOMER_ID                              40          0           9
PURCHASE_DATE                            11          0           8
PURCHASE_TIME                           492          0           4
PURCHASE_PRICE                         1032          0           5
SHIPPING_CHARGE                           3          0           4
TODAY_SPECIAL_OFFER                       2          0           2

To collect statistics for the index, CUSTOMER_GENDER_INDEX, use the following query.

SQL> EXECUTE DBMS_STATS.GATHER_INDEX_STATS('easydw',
                                         'customer_gender_index');

Statistics should be gathered after data is loaded and again whenever changes made to the data are likely to have altered the distribution. This ensures that the cost-based optimizer has up-to-date data to base its decision upon. When partitioned tables are used, only the newest partition or subpartition to which rows have been added needs to be analyzed for statistics. For example, if a new partition for July 2002 was added to the purchases table, statistics can be collected for the newly added partition, as follows:

SQL> ALTER TABLE purchases ADD PARTITION purchases_jul2002
          values less than (TO_DATE('31-07-2002', 'DD-MM-YYYY'));

SQL> INSERT /*+APPEND */ INTO purchases ...

SQL> EXECUTE DBMS_STATS.GATHER_TABLE_STATS('easydw', 'purchases',
                                        'purchases_jul2002');

To automate gathering statistics for a table enable monitoring of DML activity for that table using the MONITORING keyword in the CREATE or ALTER TABLE command. Then use the GATHER STALE option in the GATHER_DATABASE_STATS procedure to gather statistics for just those tables whose contents have changed significantly. This procedure can be called in a job that is scheduled periodically, so that statistics are kept up-to-date.

To turn on automatic DML monitoring on the purchases table, you would issue the following query.

SQL> ALTER TABLE purchases MONITORING;

Alternatively, monitoring can be turned on for an entire schema or database using the ALTER_SCHEMA_TAB_MONITORING or ALTER_DATABASE_TAB_MONITORING procedures. For instance, to turn DML monitoring on for the entire EASYDW schema:

SQL> EXECUTE DBMS_STATS.ALTER_SCHEMA_TAB_MONITORING('easydw', TRUE);

To collect statistics on only the tables that have a significant number of changes issue the following:

EXECUTE DBMS_STATS.GATHER_SCHEMA_STATS ('easydw',
                                        options=>'GATHER STALE');

7.7.1 Collecting system statistics

In Oracle 9i, the cost-based optimizer was enhanced to consider system statistics in determining the execution plan. System statistics such as CPU speed and number of I/Os per second give an indication of the resource availability in the system. This allows the optimizer to make smarter decisions when several users are contending for the same resources. System statistics can be collected using the DBMS_STATS.GET_SYSTEM_STATS procedure. As with all statistics, it is important that the system statistics give the optimizer an accurate picture of the system load. Statistics could be collected for different periods during the day-for example, during normal business hours and during after-hours reporting activities. Statistics can be saved using the EXPORT_SYSTEM_STATS procedure and later imported into a database using the IMPORT_SYSTEM_STATS procedure.

7.7.2 Dynamic sampling

Oracle 9i Release 2 introduced a new feature known as dynamic sampling, which is used when statistics on a table are absent. In dynamic sampling, the optimizer automatically collects statistics by sampling the data before optimizing the query. Dynamic sampling is useful for queries whose tables do not have statistics collected or when the statistics are too old. To enable dynamic sampling, the initialization parameter, OPTIMIZER_DYNAMIC_SAMPLING, must be set to a value greater than 1. The optimizer will perform dynamic sampling if there is more than one table in the query and some of the tables have no statistics and no indexes. Note that dynamic sampling will incur some overhead during query optimization and, hence, must be used with care. In a data warehouse, it is always a good practice to collect statistics on all your tables as part of your housekeeping procedures and to keep them current.


Parallel management tasks

7.8 Parallel management tasks

Throughout this book, the typically large size of the data warehouse means that tasks can take a very long time to complete. So far, we have considered some techniques for improving performance, but there is a very useful one that is worth mentioning again: the ability to run operations in parallel.

Normally, a single process executes a SQL statement, but imagine the performance gains we could achieve if we could split that SQL statement into a number of processes running alongside one another. Provided you are using Oracle 9i Enterprise edition, this capability is available to you.

Parallel execution is most useful for operations that access significant amounts of data, including queries, index creation, bulk inserts, updates, deletes, aggregations, and data movement. With every release, Oracle increases the number of operations and commands that can be run in parallel. At the time of writing most operations can be run in parallel, including:

  • Parallel query—queries and subqueries in SELECT statements.

  • Parallel DDL—including:

    • CREATE TABLE AS SELECT, CREATE INDEX, and ALTER INDEX REBUILD

    • For partitioned tables: ALTER TABLE MOVE, SPLIT, COALESCE

    • For partitioned indexes: ALTER INDEX REBUILD or SPLIT

  • Parallel DML—INSERT, UPDATE, DELETE, multitable insert

  • SQL*Loader and external tables

Parallel operations require accurate statistics to perform optimally. Chapter 3 discussed the use of parallelism to improve query performance, and Chapter 5 showed examples of loading data in parallel. Similar techniques are used to execute DDL statements in parallel.


Tuning the warehouse

7.9 Tuning the warehouse

One should not forget that a data warehouse needs to be tuned just like any other database, although you may be tuning for reasons such as:

  • Improving data load time

  • Improving query response

  • Refreshing data performance

  • Improving time to perform management tasks

When people think of tuning, their first thoughts are likely to be improving query performance. Traditional techniques used to resolve this problem include creating indexes and placing data in the buffer cache so that the data is faster to read. In a data warehouse, the index is still extremely important, but placing the data in the buffer cache is unlikely to help, due to the huge volumes of data typically being searched in a data warehouse.

Therefore, one of the best techniques to use to improve query performance is to create materialized views. Provided the query is used several times, if a materialized view is created that precalculates the query, the optimizer will execute the query by retrieving the results from the materialized view rather than the detail table. If your detail table is large, this can represent a very significant improvement in query response time.

7.9.1 Tuning tools

There are many tuning tools available to tune an Oracle database, from both Oracle and third-party software suppliers. If you purchase the optional tuning packs for Oracle Enterprise Manager, the following tools will be available to you:

  • Oracle Expert-creates a database design (only available on NT)

  • SQL Analyze-identifies and tunes resource-intensive SQL statements, provides graphical EXPLAIN PLAN (only available on NT)

  • Tablespace Map-identifies space management problems

  • Reorg Wizard-repairs space problems, resizes, and relocates objects

  • Outline Management-manages stored outlines

Some other useful tuning tools are in the Diagnostics Pack, including:

  • Performance Manager-shows how the database is being used, including CPU, memory, and I/O resources

  • TopSessions-displays user's database use

  • Capacity Planner-collects performance baseline statistics

Several advisors are available to give you recommendations to improve performance:

  • Summary Advisor-recommends materialized views (available with Enterprise edition)

  • Index Tuning Wizard-recommends indexes (only available on NT)

  • Buffer Cache Advisor-determines the optimal size of the buffer cache and shared pool

Just because the database is being used as a data warehouse should be no excuse to forget to tune it. However, if you are familiar with tuning OLTP systems, the techniques used will be slightly different, although the end result is the same: faster query response.

Tuning is a bit of a black art, and it is impossible in just a few pages to tell you how to tune a data warehouse. Instead, we suggest that you become familiar with the tuning tools available to you and determine what information you need to know. For instance, how much I/O was used to perform this query? Then determine how, if at all possible, you can obtain this data from your tuning tool.

Practice this technique in a controlled environment on a small system before you attempt to analyze a production system. Keep practicing to improve your confidence, and then, when you really do have to tune, it shouldn't be such an ordeal. There are many books available on Oracle tuning, and all should provide useful advice on tuning your data warehouse.

7.9.2 Plan stability

One of the significant advantages of using a relational database is that you don't have to tell the database how to access the data when a query is defined. Instead, the query optimizer determines the most efficient query execution path. Usually, the optimizer selects the best strategy, but problems can occur when new releases of the software are installed or when the database cardinality statistics change, because the optimizer may choose a different strategy. Therefore, there is a risk that your query could suddenly take much longer to complete. One solution to this problem is to use a hint, but this means amending the code, which may not be possible.

To overcome this problem, Oracle 8i introduced a new feature called plan stability. This allows you to create a stored outline for a specific SQL statement or all SQL statements. When plan stability is enabled, each query is compared with each query that has a stored outline. If the query matches a query with an outline exactly, then the strategy for the outline is used.

Creating an outline

An outline is made up of a set of hints that represents the optimizer's execution plan for a SQL statement. The hints are similar to the hints you might add to a query to influence the optimizer strategy. Outlines can either be created for all SQL statements or for specific SQL statements.

Outlines can be grouped together into categories so they can be better organized and managed. To create stored outlines for all SQL statements set the parameter CREATE_STORED_OUTLINES to true or to a category name, as shown in the following example, where the category is EASYDW_CAT.

SQL>ALTER SESSION SET CREATE_STORED_OUTLINES = EASYDW_CAT

Once you set this, outlines will automatically be created for every SQL statement. This is an easy way to create the outlines but may result in a large number of outlines. This parameter can be turned on when an application starts running and turned off by setting CREATE_STORED_OUTLINES to false when the application has finished.

Alternatively, you can create stored outlines for specific statements using the CREATE OUTLINE statement or the Outline Management Tool in Oracle Enterprise Manager, as shown in Figure 7.45.

Click To expand
Figure 7.45: Outline management.

The three outlines with names starting with SYS_OUTLINE were created automatically by enabling create_stored_outlines shown previously. Notice how outlines were created for every SQL statement-including some that are probably not needed, such as SELECT * FROM USER_OUTLINES.

The outline named SUM_OF_SALES was created for an individual SQL statement by selecting Create from the Outlines menu, as shown in Figure 7.45. The SQL for the outline is listed in the text box.

Using an outline

To use stored outlines, set the parameter USE_STORED_OUTLINES to true or to a category name. When Oracle compiles a SQL statement, it looks to see if there is an outline with exactly the same text as the query. If it finds one, then the strategy for the outline is used when executing the query.

SQL> ALTER SESSION SET USE_STORED_OUTLINES = easydw_cat

In Figure 7.45, two outlines had been used at least once, and two have not been used yet. To disable the use of outlines, set USE_STORED_OUTLINES to false.

Editing an outline

While the cost-based optimizer generates the optimal execution plan most of the time, manual query tuning can lead to more efficient plans in some cases. As mentioned previously, you could use hints, or, if you want to modify the execution strategy, Oracle 9i gives you the ability to edit an outline. To invoke the outline editor double-click on the selected outline.

Figure 7.46 shows a SQL query that computes the sum of sales and total sales by product, month, and year. To get the result, the TIME, PRODUCT, and PURCHASES tables must be joined. The strategy chosen by the optimizer is shown in graphical format. The PURCHASES table is joined to the TIME table using a hash join. The result is then joined to the PRODUCT table using a nested loop join via the primary key index, PRODUCT_PK_INDEX.

Click To expand
Figure 7.46: The original outline.

This query could also be executed by changing the join order-first joining the TIME table with the PRODUCT table and then joining the result with the PURCHASES table.

The join order can be changed from the Outline menu. Figure 7.47 shows how you can select a node in the tree and move it up or down. For the next example, the PURCHASES node was moved down.

Figure 7.47: Changing the join order.

We now have a new strategy, as shown in Figure 7.48. The TIME table will be joined to the PRODUCT table using a nested loop join and the primary key for the PRODUCT table. The result will be joined to the purchases table using a hash join.

Click To expand
Figure 7.48: Executing the query with a revised outline.

You can validate the changes and execute the query to see if it will run faster, as shown in Figure 7.48. Compare the elapsed time, CPU time, and other statistics with the execution statistics from the original strategy to see if your edits made the plan better (or worse).

If you do not use Oracle Enterprise Manager, you can edit the outlines using the DBMS_OUTLN_EDIT package. To see what outlines have been created look at the USER_OUTLINES view, shown in the following code statement.

SQL> SELECT NAME, USED, SQL_TEXT FROM USER_OUTLINES;

NAME                  USED   SQL_TEXT
-----------------------------------------------------------------
SYS_OUTLINE_020817142 USED   SELECT t.month, t.year, p.product_id
SYS_OUTLINE_020817142 USED   select * from user_outlines
SYS_OUTLINE_020817142 UNUSED select * from user_outline_hints
SUM_OF_SALES          UNUSED SELECT t.month, t.year, p.product_id

By default outlines are stored in the OL$, OL$HINTS, and OL$NODES tables in the SYSTEM tablespace in the OUTLN schema. You probably noticed this schema in your database starting with Oracle 8i. If there are a large number of outlines, they may use too much space in the SYSTEM tablespace and can be moved. Refer to the Oracle documentation for the steps to do this.

In summary, if you are concerned about strategies changing, especially as a result of a software upgrade, consider creating some outlines for your important queries. Once they have been created and stored in the database, they do not have to be used. Instead, they can be kept in case of emergency and turned on only when required. Outlines are also useful when building applications that are deployed at a number of sites to ensure that the same optimizer strategy is chosen even when there are different data cardinalities.


Maintaining security

7.10 Maintaining security

One should not forget the security of the data in a data warehouse. In many respects, warehouse data should probably be more secure than production data, because its value to your competitors could be enormous. Imagine if one of your users started earning spare cash by running reports against your data, or if an employee left the company with information from the data warehouse. If you are laughing at the last statement, please don't; although it is very rare, it actually does happen.

The level of security in your data warehouse will depend on what you think is appropriate for your system. Some sites may prefer to have the warehouse open for read-only queries, but restrict management tasks such as creating summary data to the administration team.

We have already seen in Chapter 2 how we can protect individual tables by specifying explicitly who may access the tables using the GRANT and REVOKE commands.

Another technique is to create a ROLE, assign privileges to that role, and then grant that role to a user. Roles can be created via SQL or by using the OEM Security Manager.

In Figure 7.49, a role called EASY_USER is created, and here we are specifying which tables in the database this user can access. Security isn't limited only to tables; it can be placed on a wide range of database objects. For those of you who do not want to use OEM, the SQL box at the bottom of Figure 7.50 shows the commands to implement this role.

Click To expand
Figure 7.49: Creating a role.
Click To expand
Figure 7.50: Allocating a role to a user.

Once the role has been defined, it can be allocated to a user, as shown in Figure 7.50. Roles are an extremely powerful feature in Oracle and can save an immense amount of time not having to allocate lots of individual privileges to a user.

Virtual private database

Applications that are deployed on the Internet and make information available to customers, suppliers, employees, or other users need to provide access control at a very fine level of granularity. An on-line banking system needs to ensure that customers can only see transactions for their accounts and not anyone else's account. A self-service human resources application may let employees see their own records and modify their marital status, address, and phone number, but not their salary. The same application may provide managers with the ability to read and modify the records of all employees who work for them.

As companies are increasing to focus on their core competencies, they may outsource other tasks, such as human resources, customer support, and payroll. When designing an application that provides hosting services, the data for each company must be kept separate and secure from each other.

Traditionally, access has been controlled at the object level. The data security policy determines which users have access to which schema objects and what types of actions are allowed for each object. For example, a user may be able to select from a table but not insert, update, or delete the rows in the table. If you wanted to allow a user access to a subset of rows in a table, a view could be created and the user granted access to the view. If you have a self-service application, where each employee in a large company has access to the rows containing his or her own personal data in the human resources database, each employee would have to have his or her own view. However, the number of views quickly becomes unwieldy. If you want to allow access to employee data for a certain group of users only through the human resources application, for example, and not for ad hoc queries, views do not give you this capability.

Another way to implement data security is with virtual private database (VPD) and fine-grained access control. VPD provides row-level security for all applications. Associating one or more security policies with a table or view creates the virtual private database. Any access to a table with an attached security policy causes the invocation of a function that implements the policy. The function returns an access condition in the form of a WHERE clause, which is appended to the user's SQL statement, thus dynamically modifying the user's data access. For example, in the EASYDW warehouse, we could allow each customer to see information about his or her own order history on line. When a customer issues SELECT * FROM PURCHASES, the function would add the customer_id to the WHERE clause, resulting in the following query.

SELECT * FROM PURCHASES WHERE customer_id = 'AB123459'

Often you want to control access based on some attributes about the user, such as the job code, department, location, or whether the user is a customer or partner. An application context is created to do this. Upon logging into the database, the application context is associated with the user's session. Each application can have its own application context, each having different attributes.

After creating the application context, the PL/SQL functions to implement the security policies are created. The function determines the WHERE clause to return, based on the user's application context.

The PL/SQL package DBMS_RLS is used to administer the security policies and apply them to the appropriate tables. Using this package, you can add, drop, enable, disable, and refresh the policies you create.

Many of the Oracle applications make use of VPD to provide fine-grain access control.


Monitoring space use

7.11 Monitoring space use

A very important piece of information for the DBA to know is how much space is available in the data warehouse. One technique to avoid running out of space is to create the data files with autoextend and to define an unlimited number of extents. However, this won't help you if the disk actually fills up. Therefore, as part of your routine monitoring of the data ware-house, you should check for free space. This could be done using Storage Manager in OEM, or you may prefer to simply query the Oracle metadata for this information. Figure 7.51 shows an example of the space utilization for the ORCL instance.

Click To expand
Figure 7.51: Space utilization.
Hint:

Add an OEM event to notify the DBA of approaching disk space limits.

Inside the Oracle database, there are a significant number of system tables that contain very useful information about the database. You can find a detailed explanation of all of these tables and the information they provide in the Oracle documentation.

7.11.1 Automated space management

Several enhancements have simplified space management. Locally managed tablespaces introduced in Oracle 8i eliminated the need for periodically reorganizing tablespaces to reclaim fragmented space. In Oracle 9i tablespaces can be created with SEGMENT SPACE MANAGEMENT AUTO to automate the management of free space inside a database segment such as a table or index. Automatic undo management eliminates the need to manage rollback segments manually. In Oracle 9i, Oracle managed files (OMF) can be used to create and delete files, that are needed for the data files, on-line logs, and control files, eliminating the need to directly manage the files. All you need to do is specify the location of where you would like the files stored.

7.11.2 Resumable space allocation

Long-running operations that update or add new data to the database can fail when they run out of space. Reexecuting the procedure could take a long time, particularly if it was almost complete when it failed. A new feature in Oracle 9i, resumable statements, makes it possible to intervene and correct errors in the middle of an operation. When the problem that caused the failure is fixed, the operation is automatically resumed.

A resumable statement is suspended when one of the following errors occurs:

  • Out of space

  • Maximum extents reached

  • User space quota exceeded

The following operations are resumable:

  • SELECT statements that run out of temporary space (for sort areas)

  • DML statements—INSERT, UPDATE, and DELETE

  • Import/Export

  • SQL*Loader

  • DDL statements—CREATE TABLE ... AS SELECT, CREATE INDEX, ALTER INDEX ... REBUILD, ALTER TABLE ... MOVE PARTITION, ALTER TABLE ... SPLIT PARTITION, ALTER INDEX ... REBUILD PARTITION, ALTER INDEX ... SPLIT PARTITION, CREATE MATERIALIZED VIEW, and CREATE MATERIALIZED VIEW LOG

Resumable mode must be enabled for a session. Optionally, you can specify a time-out period. If the error condition is not fixed within that time period, it will abort. You can also specify a name, which can help identify the session that has been suspended in the USER_RESUMABLE or DBA_RESUMABLE views. The following statement sets the time-out period for 3,600 seconds, or one hour, and assigns the name "data ware-house load" to resumable statements for the session.

SQL> ALTER SESSION ENABLE RESUMABLE
     TIMEOUT 3600 NAME 'data warehouse load';

SQL*Loader and the import and export utilities provide the same options as command-line parameters.

When a resumable statement is suspended, the error is reported in the alert log. The system also internally generates an AFTER SUSPEND system event. Users can register triggers for this event at both the database and schema level. The triggers can be used to notify the DBA when a statement is suspended, so corrective action can be taken.

Information about the status of resumable statements can be seen by looking at the DBA_RESUMABLE or USER_RESUMABLE views, as shown in the following example. The data warehouse load was suspended, because there was inadequate space to extend the PURCHASE_PRODUCT_INDEX.

SQL> SELECT STATUS, NAME, ERROR_MSG FROM DBA_RESUMABLE

STATUS    NAME      ERROR_MSG
--------- -----     ---------

SUSPENDED Data      ORA-01683: unable to extend index
          Warehouse EASYDW.PURCHASE_PRODUCT_INDEX
          Load      partition PURCHASES_FEB02 by 2 in
                    tablespace INDX

When a statement is suspended, the session invoking the statement is put into a wait state. A row is inserted into the V$SESSION_WAIT, as seen in the following example.

SQL> SELECT EVENT, STATE FROM V$SESSION_WAIT;

EVENT                                           STATE
----------------------------------------------- --------------
statement suspended, wait error to be cleared   WAITING

When space is added, the session will automatically resume. If space is not added within the time-out period, an error will occur.


Other management issues

7.12 Other management issues

We have already seen a number of management tasks. The ones specified here are by no means an exhaustive list, and there may very well be ones that are not mentioned here that may be applicable in your environment. There are also some additional tasks mentioned here that you may also like to consider.

7.12.1 Building a test system

When you suggest the construction of a test system, it is not uncommon for many people to throw their hands up in the air and say: Impossible. But you should stop for a moment and consider the implication on your business if you don't have a test system. When a test system exists, it can be used for a variety of reasons, such as:

  • Testing new software releases

  • Timing data loads

  • Evaluating management task times

  • Practicing management tasks

  • Testing scripts before executing in production

  • Assessing impact of maintenance tasks (e.g., index rebuild)

  • Determining query response times

Many people think that a test system has to be identical to the production system, but, in a warehouse, that is usually impossible. Therefore, what is required is a scaled-down version of the warehouse that is representative of the real warehouse. Ideally, numbers obtained from it should scale easily so that you can determine what the effect would be on your production warehouse.

Data inside the warehouse should, whenever possible, be representative of the real data. It may be that to obtain the desired effect, you may have to extract data from the real warehouse and then load it into the test ware-house.

The various uses for a test system will now be discussed.

7.12.2 Testing new software

Once a database becomes a critical component in the business and the information supplied from it is used to make critical business decisions, no one wants to jeopardize the business by introducing new software that may have software problems. Therefore, if you have a test system, you can check that all of the important parts of the database software that you use are the same. The range of items to test could be extensive. For example, you will want to check that queries use the same optimizer strategy as before. If they have changed, then you should create outlines to ensure consistency of query response times.

However, you should always ensure when using outlines that performance will not degrade when a new version of Oracle is installed—it could improve due to an optimizer change so always check for the current strategy.

Another important check to make is that key features that you rely on still function the same. For example, if you rely heavily on partition operations to maintain the data in your warehouse, then check that they still work.

Utilities you rely on should also be checked to see that no changes have occurred that cause them to change their behavior.

If a script is created that contains all of these important tasks, then each time you upgrade the software, you have only to run the scripts and check the results. Therefore, considerable effort will be required to construct the scripts the first time. Once completed, however, they can be run repeatedly and you will know that everything that is important to your environment will have been checked.

7.12.3 Timing data loads

The test system provides an ideal opportunity to determine the load time for data and to practice any data cleansing that may be required. Generally, the fastest way to load data into an Oracle 9i data warehouse is by using the utility SQL*Loader via the direct path method. But you may want to compare the performance with external tables, particularly if you also need to perform transformations.

Hint:

Don't forget to check the logs from any SQL*Loader jobs in case any problems occurred during the run, such as constraints not being enabled.

7.12.4 Evaluating/practice management tasks

Now is also an ideal opportunity to practice and try out all of those management tasks before they are done in production. When it comes to testing backup and recovery processes, using very small databases initially and then moving on to the full size once you are sure that all of the procedures are working correctly will save time.

7.12.5 Determine query response times

The test system provides an ideal opportunity to see the data warehouse in use before all the users are given access. Even with a limited user audience, you will be surprised that you will find queries that do not perform well. Therefore, you now have time to diagnose the cause of the poor performance and resolve the problem by adding a new index, for example, or creating a materialized view to make the query perform faster.

In a test environment, it is unlikely that you will see an exact reproduction of use of the data warehouse. Nevertheless, problems will still surface, and it is easier to fix them now, before the pressure comes with users demanding reports yesterday!


Summary

7.13 Summary

In this chapter, we have taken a look at some of the tasks required to manage a warehouse and used Oracle Enterprise Manager to simplify management. Ongoing tasks, including backup, gathering optimizer statistics, monitoring space usage, and refreshing materialized views were discussed. Techniques for periodic reorganization using partition maintenance operations were introduced. Developing a test system and a business continuity plan are important considerations.

This has been a long chapter, but hopefully you will now begin to appreciate what is involved in managing a database. Many of the tasks described here apply equally to a traditional OLTP database. It's often the size of the data warehouse that makes the task different. Remember that it is better to manage now than not at all. Failure to back up the database or monitor space usage could have disastrous results, usually when you can least afford the time.


Chapter 8: Data Warehousing and the Web

8.1 Overview

You may be starting to read this chapter and wonder what the Web has to do with data warehousing. One of the problems is that the Internet or Web has become a term that people use, but it has several interpretations. To distinguish between an internal Web and the World Wide Web, the term Internet is used to describe the global connection of public computers. Often, sitting behind the Internet is a company's own Web, called the Intranet, which is only accessible to people within that organization. It's the Intranet that we are primarily interested in. Most of us, when we first think of the Web, probably imagine checking out a company's product or placing an order, so what does that have to do with our data warehouse? Well, you could be very surprised at how important the Web could be to your data warehouse.

8.1.1 Internet and Intranet

Today, most companies have some presence on the World Wide Web, but what many people may not realize is that within the company there can be a huge internal Web site used by employees only, where all business is conducted. Oracle itself is a good example of a company using their Intranet to communicate and conduct internal business. For example, when Oracle employees needs a new laptop or mobile phone, they access the internal iProcurement system and go shopping, just as if they were shopping on the Web. In some countries, payslips are only available on line, as well as all end-of-year tax documents. Numerous reports, status documents, and product specifications are all accessed via Web-based systems using a browser such as Internet Explorer. Therefore, information is quickly and easily available, no matter where in the world the Oracle employee is located, provided he or she is connected to the company network. So what does a company place on its Intranet? Well, the possibilities are endless, and here are just a few ideas:

  • Press releases

  • Product information

  • Software to download

  • Payslips and HR systems

  • Purchasing systems

  • Telephone lists

  • Sales information

  • Historical data

  • Trends and analysis

  • Many different types of reports and documents

At first glance this list looks like a typical Internet site, but it's the last items, such as sales information, historical data, and trends and analysis, that differentiate it from the normal Internet and indicate a role for our data warehouse.

Before we look at how we can use this Intranet, a concern many people have when data is published on the Web, even if it is internal, is whether or not this data is safe. Typically, company Intranets are not connected to the Internet and can only be accessed by either being in a company office, by dialing in, or through a secure VPN connection when outside a company office. Therefore, the information is safe and protected from hackers. Of course, if you are concerned about security within your organization, information can be protected so that only certain staff can review sensitive information.

Therefore, knowing that our data is safe, why would we want to consider putting our data warehouse on the company Intranet? Well, before the Intranet, if there was a report a manager needed, say sales by area for this month, the manager would have to request that report and then wait for the paper copy to arrive on his or her desk.

Now, the Intranet changes all of that, because standard reports can be created and then published on the Intranet for managers to review at a time that is convenient to them. Reports can be made readily available to everyone, no matter their location, once they have been produced. No longer do you have to wait for a report to arrive in the internal mail. If you are not in the office today but working from a hotel, home, or dialing in from some location, you can still review the report. You could even consider converting the report into a spreadsheet so that it can be downloaded and manipulated locally.

Although the report could be run and simply published on the Web, ideally our users should also be able to create the report using tools such as Discoverer Viewer and Discoverer Plus, which are described in Chapter 6. Using this approach, not only can our user get the report when they want it, but they are no longer restricted to getting a standard report. Because now tools such as Discoverer allow then some degree of customization of the report, such as only sales for the southern area or only a store whose sales are less than $10,000.

Hopefully, now you are beginning to appreciate how the Intranet can help solve your report publishing and distribution problems and provide a mechanism for your users to have access to the very latest information. For example, suppose you have a static report. You could automatically regenerate that report every hour and publish it on the Web, or refresh it on demand as users request it. On-line interaction with reports has traditionally not been feasible with a data warehouse, because queries can take hours to run. But now, with features such as Summary Management, described in Chapter 4, a query that used to take hours can be reduced to seconds or minutes. Thus, the report using that data can be run more frequently.

Now that we have an appreciation of what the Web can do, there are several different ways in which it can interact with our data warehouse:

  • It becomes the deployment platform for all your business intelligence information using tools such as Oracle Discoverer and Reports.

  • It can also be a source for our data warehouse by tracking activity on our Web site-this is known as clickstream data.

  • It becomes the platform where an application can be customized using Oracle Personalization, thanks to data from our warehouse.

8.1.2 Oracle software for the data warehouse

Oracle offers a number of products that can help you publish the data from your data warehouse on your Intranet or Internet, understand how your Web site is being used, or use the access visitors made previously to give them a customized experience. The components that are of interest to us include the following:

  • Oracle Reports

  • Oracle 9i Application Server, which includes:

    • Discoverer

    • Portal

    • Clickstream Intelligence

    • Personalization

We will now look briefly at each of these products and see how they can help our data warehouse.


Oracle 9i Application Server

8.2 Oracle 9i Application Server

Hopefully, you are now beginning to appreciate how the Web can dramatically change how information is used and processed by the data warehouse. Although not a prerequisite, using Oracle 9i Application Server (9iAS) to run all of your business intelligence applications provides you with the complete environment to run e-business intelligence.

Oracle 9i Application Server provides in a single product all the technology you need to build and implement e-business portals, Web services, and transaction-based applications. It supports all the Java, XML, and Web services industry standards and can be used to access both Oracle and nonOracle data.

Included is Oracle's HTTP Server, which is based on the Apache Web Server. It has already been configured and is immediately ready for use. There is also a J2EE (Java 2 Enterprise edition) environment, which is a Java platform designed for the development of applications up to Enterprise size businesses. Using J2EE technologies, you can create an application using Java server pages, servlets, and Enterprise JavaBeans. A Java servlet is a program that, when it receives a request for information from the client, dynamically gets the information from the database and generates a response that can be in HTML or XML and sends that back to the client. JavaServer pages help us develop our servlets, because they allow the developer to easily design and maintain dynamic Web pages by separating the dynamic content from the page layout. Enterprise JavaBeans help the developer by encapsulating business logic, so there is no need to write the code for these functions.

Within Oracle 9iAS you can do development using Java, XML, Perl, and PL/SQL. Oracle 9iAS actually comprises the following products, although you don't have to install all of them.

  • Oracle HTTP Server

  • Oracle 9iAS Reports Services

  • Oracle 9iAS Portal

  • Oracle 9iAS Discoverer

  • Oracle 9iAS Clickstream Intelligence

  • Oracle 9iAS Personalization

  • Oracle 9iAS Forms Services

  • Oracle 9iAS Wireless

  • Oracle 9iAS Web Cache

  • Oracle 9iAS Unified Messaging

  • Oracle 9iAS Security

  • Oracle Internet Directory

  • Oracle 9iAS Containers for J2EE

  • Oracle Enterprise Manager

Now we will begin to see which components you may need to take your data warehouse to the Web. There are many ways that a Web site can be configured, and it is beyond the scope of this book to discuss them here, but typically the Web server will be running on a machine different from the data warehouse.

8.2.1 Why set up a portal?

When the Internet first became popular, a company would talk about its Web site, but now it talks about its portal. A portal is a Web site that provides a lot of information and services. If a Web site is devoted to a specific topic, such as the medical site WebMD (http://www.webmd.com), it is known as a vertical portal. But most portals cover a wide range of subjects, such as Yahoo (http://www.yahoo.com) and are known as horizontal portals.

The portal provides access to this wealth of information, and typically upon your arrival, once you sign in, what you see is customized to meet your needs. For example, when you visit Oracle (http://myoracle.com), shown in Figure 8.1, you will see a Web page completely different from what an Oracle employee will see.

Click To expand
Figure 8.1: Signing into the Oracle portal.

The Oracle employee will be presented with internal company information and access to a wide range of internal systems. Whereas the typical visitor to the Web site will first be presented with the screen shown in Figure 8.2, where visitors are invited to register for Oracle World and are presented with the latest news. Since this is a portal, visitors can now customize this page so that they only see and access information that is of interest to them.

Click To expand
Figure 8.2: Oracle portal—standard view.

So why is a portal relevant to our data warehouse? Well, many companies are now realizing that they need a portal, rather than a Web site, because an internal portal can make it much easier for employees to function, since they have customized access to the information they need to do their job, thus increasing their efficiency. Therefore, consider using a portal to bring the information from your data warehouse to its users. There are two products in 9iAS that make this possible: portal and personalization.

Oracle 9iAS Portal

Oracle 9i Application Server includes Oracle 9iAS Portal, which is software that allows a portal to be developed quickly and easily. Since 9iAS Portal uses the single sign-on authentication that is part of Oracle 9iAS, it provides the mechanism by which access to the information is controlled.

It's very easy to develop applications using 9iAS Portal, even if you are not an experienced developer. It includes a number of wizards, and these can be used to quickly create reports and charts that our users can view. When data is being retrieved from the database, there are also wizards to help you specify the query, or you can define the SQL yourself. The benefits of using 9iAS Portal include the following:

  • It has a framework that is extensible, which integrates all the Web resources.

  • It uses the 9iAS single sign-on.

  • It provides easy access to information.

  • The contents can be personalized.

  • Wizards build applications quickly and with minimal development knowledge.

  • It uses a scalable architecture.

  • It is an integrated solution with Oracle 9i Reports and Discoverer.

  • It publishes information using portlets.

  • It is a self-service publishing facility.

The self-service publishing feature is of interest to our data warehouse users, because it allows them to publish and share documents. Therefore, we can use the portal to create a portlet—that's one of those tabs in Figure 8.2, where we can publish the information from our data warehouse. Using this approach to publish and retrieve information in our data warehouse means that any user within our organization who has permission to access the data can easily view the data. They simply click on the tab and are taken to the report or graph they are interested in.

Although Oracle 9iAS Portal can create simple reports, it is also tightly integrated with Oracle 9i Reports, which we will discuss later in this chapter. Using a wizard, a report created in Oracle 9i Reports can be registered in Oracle 9iAS Portal and published as a portlet. Also, the simple reports created using Portal Reports can also be converted into an Oracle 9i Report. However, since only simple reports and charts can be created within Oracle 9iAS Portal, most people will want to use the sophisticated reporting capabilities in Oracle 9i Reports and then turn that report into a portlet.

With Oracle 9iAS Portals customized pages, users have tremendous control over how they run and see the report. For example, if the report has been set up to accept parameters such as a date and store, users can ask for the data they are interested in, such as only the stores in their area for the past four weeks, and see that version of the report in the portal. Another user could request different data and see his or her own customized version of the report.

Once in the portal, the report contents do not have to remain static. It can be automatically refreshed and republished with the new data, and it is even possible to keep multiple versions of the report, should this be a requirement. Therefore, we can see that the tight integration between Oracle 9iAS Portal and Oracle 9i Reports can provide an incredibly powerful medium for making available the information in our data warehouse.

So far, we have seen how we can use the portal to easily present and retrieve information, but wouldn't it be nice if we could extend this further and customize the information being presented to each visitor to our portal? This can be achieved using Oracle Clickstream and Personalization.

8.2.2 Oracle Personalization

Have you noticed that when you visit some Web sites they remember who you are and then the next time you visit the site, they start recommending specific products, making you aware of specific special offers they think you might be interested in, and even tailoring the products they show to suit your needs? If you have ever wondered how they do it, the answer is through products such as Oracle Personalization and your data warehouse.

What Oracle 9iAS Personalization does is take the information stored in the data warehouse about what customers do each time they visit your Web site. Then, using the data mining technology that is built into Oracle 9iR2, it makes recommendations to customers as to what it thinks they will be interested in. Using Oracle Personalization questions such as the following can be answered:

  • Which product will our customers mostly likely buy?

  • What other items may be bought when they buy this item?

  • What rating may a customer assign to a product?

  • Which items on the hot pick list do we think our customers will purchase?

Once again, we are using all of the information in our data warehouse not only to benefit the customer but hopefully to benefit our business both financially and with respect to customer service.

8.2.3 Clickstream intelligence and the data warehouse

Traditionally, when people think of a data warehouse, they envisage lots of historical data (e.g., all of the telephone calls made during the last year, all the products that were sold, or all of the staff employed over the last ten years). But now we should be looking at our Web site for lots of data that is ideal to be stored inside the warehouse. This data may represent activity over quite a short period of time.

However, even though it may be for a short period of time, there could be a huge amount of data here, because with clickstream data, we are monitoring what the visitors to our Web site are doing. Therefore, if 250,000 people visit our Web site today and they all look at five pages, that could be a lot of information to store. So, another area where our data warehouse can get involved with the Internet is to store all of this clickstream data—that is, what people did when they visited our Web site.

When you run a traditional brick and mortar business, you can easily watch your customers as they peruse your store. You can see them walk in, pick up an item, put it back, and walk out. Or they may go directly to the item and then go to the checkout, or walk up and down the aisles to see what you have today before they go to the checkout. For an e-business, how do you find out what your customers aare doing, since they are probably at home, and you are in your office, maybe thousands of miles away.

Products such as Oracle Clickstream, which is part of Oracle 9iAS, monitor what people do when they visit your Web site. It has been integrated with Oracle 9i Warehouse Builder and Oracle 9iAS Discoverer to enable a complete e-business intelligence system. Now you can identify which pages customers visited, what they added to the shopping cart, and whether or not they waited for pages to load.

There is a wealth of information here that can be used to help improve your business. What better place to store and analyze this data but in the data warehouse? Why a data warehouse? Well, if you are a well-known airline receiving 600,000 visitors a week to your site, you will accumulate a lot of data; this data must be stored if it is to be saved for any period of time. In the past, data warehouses recorded only data about an event that happened, such as the purchase of an item. Now is the time to realize that other types of data can be stored in the data warehouse.

By now you should begin to understand the important role the Web can play in bringing the information contained in our data warehouse to the people who need it. Now we will look at some of the tools that enable us to see this information from our browser.

8.2.4 Oracle Discoverer Reports via the Web

There are a number of tools that can be used to retrieve information from our data warehouse, and a very popular one is Discoverer, because it provides companies with a mechanism for easily accessing the data in their data warehouse and integrating that information with Oracle 9iAS Portal so it can be presented via a browser. Alternatively, you can use Oracle Discoverer Plus or Viewer, which are part of Oracle 9iAS, and they also allow you to run the reports we created in Chapter 6 from your browser.

Using Discoverer Viewer or Plus

In Figure 8.3, we can see one of these reports as executed and viewed by Discoverer Viewer from our browser. Now, in this workbook we have three reports: Total Sales by Customer, Best Customers, and Most Popular Products. Each one is presented as an active link on a tab on the left side of the screen. Therefore, it's very easy for our users to select the report they require.

Click To expand
Figure 8.3: Discoverer Viewer Report from our browser.

Although we are using our browser to display the data, the report does not have to be static, since we can interact with the person running the report. If the Most Popular Products report is selected, the screen shown in Figure 8.4 appears, where we are asked the criteria that determine the popularity of a product. In this example, we have said that we are interested in any product where we sold more than 100 units.

Click To expand
Figure 8.4: Discoverer Viewer—specify reporting criteria.

Discoverer Viewer provides an easy way to retrieve information from your browser. In Figure 8.3, there is some customization of the report that is possible, but when using Viewer, the user can only read the data and there is no possibility that this data can be altered.

Distributing reports to the Web

Discoverer Viewer or Plus are actually running the report via our browser, but what if we just want to publish the report on the Web so that it can be viewed as a static document? This is also possible, and, referring to the workbook where we first created the reports in Chapter 6, this was done using the non-Internet version of Discoverer. But once in Discoverer Plus, we can take any of those reports and share them with other users by exporting the worksheet to a number of formats, including:

  • XML for Oracle Reports

  • Excel

  • HTML, so it can be e-mailed

  • Comma-Separated Values (CSV)

The process for exporting a worksheet or workbook is very straightforward, thanks to the Export Wizard, which is started from the File menu. Simply select which worksheets you want to send to Oracle Reports, and then select whether just a table or a table and a graph are required. Next we must specify the format of our report; we can see those options in Figure 8.5, where a drop-down list of formats is provided. Discoverer Reports to be published on the Web should be exported in Oracle Reports format.

Click To expand
Figure 8.5: Discoverer Plus—Export Wizard—specify output format.

All we now need to do is specify the location for our report; the final screen, which has been omitted from this book, merely asks if the export is to run supervised, which gives you the opportunity to fix any problems that may occur, or unsupervised. Click on the Finish button, and your report is now ready for Oracle Reports.

Reports can also be integrated from Discoverer into Oracle 9iAS Portal, which provides an extremely effective way to deliver customized data to the end users. This is achieved by Oracle 9iAS Discoverer being a portlet provider to Oracle 9iAS Portal. There are two types of portlets that Discoverer can provide:

  • List of workbooks portlets

  • Worksheet portlets, which is an individual worksheet

One of the really nice features is that when the page for the portal is being designed, there can be as many portlets on that page as are needed. Therefore, multiple Discoverer worksheets and graphs can appear on a single portal page. When the user accesses that page, he or she can either select which worksheet is needed by clicking on it, or a user can have worksheets and graphs automatically displayed on the page.

Our end user doesn't even have to worry about whether the data being retrieved via the portal are up-to-date, because when the portlet is designed, you can specify how frequently the portlet should be refreshed. With users being able to customize the Discoverer portlet to their own requirements, using Oracle 9iAS Portal to present Discoverer reports and graphs provides a friendly, easy to use, customized environment.


Oracle 9i Reports

8.3 Oracle 9i Reports

Previously, we have seen how reports can be generated using tools such as Oracle Discoverer. But if you want to create sophisticated reports, using data from a variety of sources, which can then be published in a paper or an on-line format such as HTML or PDF, consider using Oracle 9i Reports.

Oracle 9i Reports will accept data from a variety of sources, including, an Oracle database, Oracle Express, 9i OLAP, XML, JDBC, or even a simple text file. For example, Discoverer uses XML to send a report to Oracle Reports. The resulting report can be presented in a variety of styles and can be enhanced by adding graphs such as pie charts or bar charts. Each report can be based on predefined templates, or the Template Editor can be used to create your own templates, which means that you can define a standard layout and include, for example, your company logo. Once the report has been defined, it can then be sent to a number of different destinations, including a file, a printer, e-mail, or Oracle Portal.

Oracle Reports is composed of several components:

  • Report Builder: a Wizard-based tool used to create reports

  • Oracle 9iAS Reports Services

    • Reports converter to change the format of a report

    • Queue Manager

Let us now look at some of the types of reports we can produce using Oracle Reports.

8.3.1 Creating a report using the Report Builder

A report can either be constructed manually or by using the Report Wizard, which will guide you through the steps of creating a report. In Figure 8.6, we see one of the first screens, where we are asked to select the type of layout. Since our report is going to be published on our Intranet and distributed in hard copy, we have selected the Create both Web and Paper Layout option. This means that a single report can have two formats if required—one for paper and another for the Web.

Click To expand
Figure 8.6: Oracle Reports—select the reporting medium.

Our next step is to select the type of report we require, as shown in Figure 8.7, where you can see there are a number of options to choose from; a thumbnail sketch of the style is presented next to each radio button. The title of this report will be "Sales by Customer," and the data will be presented in tabular format. Later we will see examples of reports using the matrix option.

Click To expand
Figure 8.7: Choosing the style of report and title.

In Figure 8.8, we can see the ways that Oracle Reports allows us to query the data source. In this example, we have chosen to use SQL, but it could just as easily be a query to Oracle Express or JDBC.

Click To expand
Figure 8.8: Oracle files—selecting the data source.

Since we are querying the database, we see in Figure 8.9 that there are three methods available for defining the SQL. If you know SQL, then you can type it in manually, or the SQL can be imported from a file. In the example in Figure 8.9, the SQL statement was created using SQL*Plus, saved in a file, and then imported into the Report Builder.

Click To expand
Figure 8.9: Oracle Reports—specifying the SQL query.

For users not familiar with SQL, use the Query Builder, because it makes defining SQL statements very easy and will certainly save you a lot of time. All you have to do is select the tables to include in your report and then tick the items that are to appear. Oracle Reports automatically determines how to join the data and creates the required SQL when you leave the Query Builder. In Figure 8.10, we have selected the CUSTOMER, PRODUCT, and PURCHASES tables and only asked for four columns from all of these tables. Note how Oracle Reports has determined how to join the PURCHASES table to the CUSTOMER and PRODUCT tables.

Click To expand
Figure 8.10: Oracle Reports—using the Query Builder.

Oracle Reports will now validate our SQL statement. Next, we are asked what grouping is required. A report with a grouping can contain one or more levels of subtotals, but in Figure 8.11 there will be only one: on name.

Click To expand
Figure 8.11: Oracle Reports—specifying the grouping levels.

The columns that will be printed in the report are selected next—all the columns in the query or a subset of them may be chosen. In the example in Figure 8.12, we will be displaying our customer's name and the total purchases by customer.

Click To expand
Figure 8.12: Oracle Reports—selecting the fields in the report.

We can now select, in Figure 8.13, whether we require any totals to be computed for data. See how Oracle Reports gives us buttons to request sum, average, count, minimum, maximum, and %total. In our example all we need is the sum of purchase_price.

Click To expand
Figure 8.13: Oracle Reports—calculate totals.

A nice feature in Oracle Reports is the ability to specify how wide the columns should be for our data and what our column headings should be. In Figure 8.14, we are given the option to specify these widths before our report is produced. Therefore, we have taken the opportunity to increase the width of the customer name to 20 characters and change its heading to Customer.

Click To expand
Figure 8.14: Oracle Reports—specify the column widths.

The layout for the report can be determined by selecting one of the predefined templates, or you can define your own template. When you're done, click Finish and you are returned to the Reports Builder. Once here you can save the report and then select Program followed by Run Web Layout to actually run the report. Figure 8.15 shows us the actual report. However, this is in its raw format—before it is customized with company logos and other items that are required. From within Reports Builder you can also modify your report by adding currency symbols, commas, and decimal points. You can also change the typeface and font (bold, underline, italics) of text. You can format numbers to represent a monetary value and add your own logos, graphics, headings, and footers. It is indeed a very comprehensive reporting tool.

Click To expand
Figure 8.15: Oracle Reports—final report.

8.3.2 More Oracle Reports examples

The report shown in Figure 8.15 is a very simple report and not that exciting, but it illustrates how to create a report. Let us now look at some examples of more reports we have created using Oracle Reports from our EasyDW data Warehouse.

Matrix report

In Figure 8.16 we see an example of a report using the matrix layout, where we can view the total amount our customers have spent by day. Looking at this report we can see that our January 1 sales promotion created a lot of business for us, but after that business dropped off and some customers did not buy from us again for another few months.

Click To expand
Figure 8.16: Oracle Reports—matrix report.

Conditional report

The extensive capabilities of Oracle Reports begins to become apparent when we see how defining a condition on our report can be used to highlight when specified conditions have been reached. In Figure 8.17 we have chosen to highlight in red any customer who spent less than $250. We could even go further and specify a number of conditions, all highlighted with a different color. Therefore, customers spending less than $250 are highlighted in red, up to $1,000 in yellow, and over $1,000 in black. Unfortunately, we can't show this, so the red numbers are in italic.

Click To expand
Figure 8.17: Oracle Reports—conditional report.

8.3.3 Building a report for the Web

Everything we have seen so far can be used to publish our report on the Web. One of the nice features in Oracle Reports is that you can design your report with one format for the Web and present it in a different format for paper. We can see the two formats in the Object Navigator window shown in Figure 8.18.

Click To expand
Figure 8.18: Oracle Reports—Navigator and Designer.

Publishing this on the Web can be done in various ways, such as making it available to Oracle 9iAS Portal as a portlet or turning the report into an HTML document. We can turn our document into HTML or PDF format either by the File then Generate option or by using the Oracle Reports Server. In using Reports Builder, as shown in Figure 8.18, click on File and then Generate to File and select the output format; in this case we have chosen a paginated HTML, but we could have selected PDF. Our report is converted and saved in HTML format.

In Figure 8.19, we can see how our report looks in our browser, which is exactly how we designed it in Figure 8.18. Note how we have added a date to the report; changed the format, layout, and fonts for the heading; removed the logo; displayed our total in red; and added some blank lines underneath the total.

Click To expand
Figure 8.19: Oracle Reports in Web format.

Oracle Reports Server

If the Oracle 9iAS Reports Server is started, a report can be exported into reports run-time format and placed on the queue, where it is converted into one of the following formats:

  • ASCII

  • HyperText Markup Language (HTML)

  • HTML Cascading Style Sheets (CSS)

  • Adobe Portable Document Format (PDF)

  • XML

From the Oracle Reports Queue Manager, a new job is specified and the screen shown in Figure 8.20 appears. Here you specify the report to be converted, which in our example is Sales_by_month, and then the name of the output file, sales_by_month.pdf, since we are making an Adobe Acrobat file.

Click To expand
Figure 8.20: Oracle Reports Server—select the report.

As you can see, there are a number of options that can be specified. You must select the Data Source tab to specify the database, user name, and password to be used to retrieve the data. When we click on the Output Options tab, the screen shown in Figure 8.21 appears, where we specify the output format; this is where we have chosen the Adobe Acrobat format of PDF.

Click To expand
Figure 8.21: Oracle Reports Server—specify the output format.

By selecting PDF and HTML formats, you have the ability to view the reports using a Web browser. Oracle Reports also allows you to deploy a report to:

  • File

  • Printer

  • Oracle Portal

  • E-mail

  • Cache (for direct output in the browser)

The advantage of using the Oracle Reports Server is that a report can be scheduled to be updated on a regular basis. In Figure 8.22, we have selected the Scheduling Options tab, where we have asked that our report be refreshed every hour. Once the report has been created, it can then be distributed or placed in our portal or Oracle files for users to read.

Click To expand
Figure 8.22: Oracle Reports Server—schedule a report.

When we have finished defining our job, it is automatically placed on the reports queue. When it is time for it to be run, the appropriate conversion will take place. In Figure 8.23, we can see our report, called Sales_by_month, which completed successfully and generated our PDF file.

Click To expand
Figure 8.23: Oracle Reports Server—job queue.

Now that we have our report, it can be distributed via e-mail, Oracle files, or through the portal. Hopefully you agree that there are many easy to use tools available to make this task quick and easy.


The data warehouse and e-business intelligence

8.4 The data warehouse and e-business intelligence

The Internet and company Intranets are creating huge opportunities for organizations. For businesses they are identifying new customers and encouraging existing customers to switch their loyalties to other companies. Within a company they are providing a flexible, easy to use, working environment. The data warehouse can be a key component in making your ebusiness successful. Why? Because it can help answer all those important questions that ensure your business succeeds, such as:

  • What products people buy

  • When these products are sold—time of year and time of day

  • When there is a quiet time when systems can come off line

  • What we predict our sales will be this time next year based on current sales

  • How much free shipping will cost us

Hopefully, this chapter has given you an insight into how you can use the Web with your data warehouse. There are many ways it can be exploited and we have barely touched the surface here with respect to the tools and techniques that are available. Oracle is also continually enhancing the tools to facilitate easier construction of Web-based applications, and new announcements are being made all the time, so keep visiting Oracle's Web site at http://www.oracle.com to get all of the latest information. If you have never experienced customizing a Web site, then try it for yourself at http://myoracle.com.


Chapter 9: Oracle 9i OLAP and Data Mining

No book on data warehousing would be complete without a discussion of its two important applications—OLAP and data mining. In Oracle 9i, OLAP and Data Mining functionality are now integrated into the database server and are available as options to the Enterprise Edition of the database. We will discuss each of these options in this chapter.

9.1 Oracle OLAP

Online Analytical Processing (OLAP) is used to answer queries that provide information about your business. OLAP queries typically perform complex calculations, such as period-over-period comparisons, finding top-N products, time series analyses, and sales forecasting. The data is analyzed along multiple dimensions, such as product, geography, and time. Hence, this type of data is often called multidimensional data.

ROLAP versus MOLAP

Ever since the OLAP industry started, there has been an ongoing debate about the best way to store multidimensional data. One school of thought advocated storing data in tables in relational databases, which are known for their ability to scale to large amounts of data. This is known as Relational OLAP (ROLAP). In this case, analysis of data is done using SQL queries—all of the SQL features discussed in this book would be useful for relational OLAP. The other school of thought felt that multidimensional data processing should be done using a specialized storage format (called a multidimensional database or MDDB) designed to quickly answer OLAP queries. This is known as Multidimensional OLAP (MOLAP). Many vendors provide MOLAP capabilities, including Oracle's own Express Server product. The benefit of these products is that users can access the data very easily without complex queries. However, a major problem with this approach is that data has to be moved from the relational database, where it is typically stored, to the multidimensional database for analysis. This means that the data is replicated and can get out of sync. Further, MOLAP products, being primarily focused on analysis do not provide the same level of security and reliability that a relational database does. Having said that, both mechanisms have their merits, and whether the right choice is ROLAP or MOLAP depends on the application in question. In fact, prior to Oracle 9i, most businesses would have had to invest in both technologies—a relational database for simple analysis and reporting needs and a specialized analysis product for advanced analysis and business planning applications. Oracle 9i eliminates the need to have two separate analysis products by allowing you to do both within the Oracle database server.

Oracle 9i OLAP

Over the past few years, starting with Oracle 8i, Oracle has been incorporating OLAP functionality into the database to support relational OLAP. Analytical functions and the aggregation operators, CUBE, ROLLUP, and GROUPING SETS, discussed in Chapter 3, allow users to do complex OLAP calculations through SQL. Materialized views and query rewrite allow you to preaggregate data so that queries can be answered quickly.

With Oracle 9i Release 2, Oracle now also supports multidimensional OLAP directly in the database. This is available via the OLAP Option of Oracle 9i Database, Enterprise Edition. With the OLAP Option, the database can store data in a multidimensional format in an entity known as an analytical workspace. Further, there is a full-featured multidimensional calculation engine built into the database. This means that you have the full analytical capabilities provided by any traditional MOLAP product. However, with Oracle 9i OLAP Option, you also get the added benefits of security, manageability, and reliability provided by a database management system. Thus, Oracle 9i Release 2 can be called a relational-multidimensional OLAP database!

With the introduction of the Oracle 9i OLAP Option, the calculation capabilities of Oracle's MOLAP product, Oracle Express Server, are now available in the database. Existing Oracle Express databases can be migrated to analytical workspaces in the database. You can build Java applications for OLAP using OLAP APIs and reusable components known as BI Beans, integrated into JDeveloper. This is analogous to the Oracle Objects application development environment supported by Oracle Express, except that it is a lot more powerful. In the future, products such as Oracle Financial Analyzer and Oracle Sales Analyzer, which use Express, may be replaced by applications that use the Oracle 9i OLAP functionality.

Oracle 9i OLAP Option and Discoverer

You may wonder why you need the OLAP Option when you already have tools such as Discoverer. Discoverer and the OLAP Option serve complementary functions. Discoverer is a packaged application that is suited for ad hoc querying and reporting needs. It is designed for end users who may not know SQL. However, some applications and users require more complex analytical functionality, such as financial forecasting, budget allocation, and modeling. The OLAP Option is targeted toward these high-end analytical needs. Further, the OLAP Option also provides an OLAP application development framework. With BI Beans and OLAP API, you can write custom applications specific to your business requirements.

We have discussed Oracle's SQL query processing features at length throughout this book. In this chapter, we will focus on the multidimensional analysis model provided by the OLAP Option. For those of you who have used Oracle Express before, we will point out the equivalent concepts in the two products wherever possible.

Let us start by looking at some of the applications where the OLAP Option is most useful.

9.1.1 Analytical applications

Once a data warehouse has been built, a business may deploy a host of business intelligence applications. Some of these may be ad hoc querying and reporting applications such as Discoverer and Reports, discussed in Chapter 6. However, a business may need several advanced analysis applications, such as demand planning, sales forecasting, corporate budgeting, and financial modeling. These applications not only analyze data but also generate new data for the business. The Oracle OLAP Option is very well suited to these advanced analytical applications, since it provides a multidimensional storage and analysis model. Let us briefly look at some of the operations commonly performed by these applications.

Analytical calculations and aggregation

OLAP applications often require aggregation and analyses involving ranking (such as top-10 products), time-series calculations (such as moving average), and inter-row calculations (such as period-over-period comparisons).

Note that these calculations can now also be done in SQL, as discussed in Chapter 3, but have traditionally been done by multidimensional analysis tools.

Forecasting

Forecasting, as the name suggests, involves predicting a quantity based on available historical figures—for instance, forecasting sales for the next quarter based on results of the past year. These applications use advanced statistical algorithms, such as linear and nonlinear regressions, single and double exponential smoothing, and Holt-Winters method.

Allocation

Allocation, also known as reverse aggregation, is used to divide a quantity such as a budget or a quota into several parts. Allocation is an important part of business planning applications. These applications require the ability to apportion a budget within a hierarchical structure or among specific members in the organization. The allocation may be weighted using user-defined parameters or analytical functions.

Financial calculations

These are calculations that can be conveniently done in a spreadsheet environment, for example, interest calculations and payment schedules.

Modeling

Modeling involves describing a quantity using a set of equations. The model can then be used to compute other quantities by plugging in data into these equations. The equations may have an implied dependency order among them and can compute new values of dimensions and facts. For example, a simple modeling program may involve equations such as the following:

Profit   =  Revenue - Cost
Revenue  = SalesChannel('Internet') + SalesChannel('Retail')
RetailSales('Clothes') = RetailSales('Mens wear') +
                         RetailSales('Womens wear')
RetailSales(quarter) = lag(RetailSales,quarter),1) * 1.1
...

What-if analysis

What-if analysis or scenario management is a very important aspect of advanced analytical applications. It involves analyzing data under hypothetical scenarios to determine their impact on the business. For instance, how much will it cost the company if we were to close down some of our retail stores and start an online outlet store? What will be the impact on revenues if we made a change to our sales organization? What-if analysis requires a transactional model different from that provided by traditional SQL. Users must be able to change the structure and content of the data in a localized manner without making the data visible to the entire database. Further, the changes may be temporary and the user should be able to restore the data back the way it was. You can think of this as a long-running session where you do a sequence of DDL and DML operations and commit and roll back the changes; they are visible only within your session.

Oracle 9i OLAP supports all the above operations with the 9i OLAP Option.

9.1.2 Oracle OLAP architecture

We will now describe the various components that form Oracle 9i OLAP and then discuss how to use each of them.

Hint:

There has been a significant change in the architecture of Oracle OLAP in 9i Release 2. All examples and functionality described in this chapter refer to 9i Release 2.

Figure 9.1 shows the architecture of Oracle 9i OLAP. It consists of the following components:

  • Oracle 9i database and SQL engine

  • Analytic workspaces

  • OLAP calculation engine

  • OLAP DML command language

  • SQL table functions

  • OLAP catalog

  • OLAP API

  • BI Beans

Click To expand
Figure 9.1: OLAP 9i Release 2 architecture.

Oracle 9i database

The Oracle 9i database serves as the data store for all data in the OLAP Option. The data may be stored in relational tables (typically in star or snowflake schemas) or in multidimensional format in analytical workspaces within the Oracle database. Consolidating all types of data into one data store provides many advantages:

  • All data can be managed by a single application, Oracle Enterprise Manager.

  • The database provides a uniform security mechanism for all types of data.

  • The data does not have to be moved into a separate database and hence the time lag involved in making data available for analysis is reduced.

  • The reliability and availability of the data are improved due to features such as Real Application Clusters and Data Guard.

Analytical Workspaces

An analytical workspace is an entity that stores data in a multidimensional format within the Oracle database. As with any other database object, an analytical workspace has a specific owner. However, unlike other objects, an analytical workspace can be permanent or temporary for the duration of analysis. You would use an analytical workspace for data when you need to use the advanced analytical capabilities of the OLAP calculation engine.

OLAP calculation engine

The OLAP calculation engine is a full-featured multidimensional calculation engine running inside the Oracle 9i database server. It operates on data stored in multidimensional form in analytical workspaces. It complements the analytical features provided by SQL with various calculation capabilities. Some of the common types of analyses done using it include forecasting, allocation, and modeling.

OLAP DML

OLAP DML is a simple but powerful programming language that is used to load, manipulate, and query data in analytical workspaces. It provides several operations, such as aggregation, forecasting, regression analysis, numerical calculations, and time-series manipulation. OLAP DML can be issued either by using the OLAP API, the DBMS_AW PL/SQL package, or the OLAP Worksheet tool (similar to SQL*Plus) in Oracle Enterprise Manager. If you are familiar with Oracle Express, you may recognize that OLAP DML is very similar to the SPL language. In fact, existing programs written for Oracle Express should work with only minor changes in OLAP DML.

Table functions

In Chapter 5, we saw how table functions can be used to perform various ETL functions for your data warehouse. A table function can perform any kind of computation underneath but finally produces its output as a set of rows. Hence, it can be used in SQL queries as if it were a table in a database. Because of this capability, table functions provide a mechanism to access multidimensional data using SQL. If you have a table function that provides access to the analytical workspace using OLAP DML, applications can then use it like a table in a regular SQL query. Oracle provides a table function, called OLAP_TABLE, that provides SQL access to an analytical workspace. This makes Oracle OLAP accessible to users who are unfamiliar with OLAP DML.

OLAP Catalog

The OLAP Catalog describes the logical metadata model for multidimensional data, using dimensions, levels, hierarchies, attributes, measures, and cubes. You should be familiar with the terms dimension, levels, attributes, and hierarchies from the SQL dimension object, discussed in Chapter 4. A measure is the same as a fact in your fact table, such as purchase_price, and is typically used in aggregations. A cube defines how measures are computed along various dimensions. Later in this chapter, we will describe how you can create these entities using Oracle Enterprise Manager. Note that the OLAP catalog merely defines the metadata and does not actually populate the data into the cube. You must create metadata in the OLAP catalog if you would like to use OLAP APIs, BI Beans, or create analytical workspaces using Oracle-supplied packages.

OLAP API

The OLAP API is a set of Java programming interfaces for Oracle OLAP. The OLAP API allows application developers to write programs to perform calculations and multidimensional selection and navigation through the data. Since it is Java based, the OLAP API provides a portable, object-oriented application development framework for OLAP applications. The objects being manipulated by OLAP API must first be defined in the OLAP catalog.

BI Beans

BI Beans are reusable components specially designed for rapid development of OLAP applications. You can create BI Beans using simple wizards in JDeveloper and store them persistently in the database. BI Beans can perform various operations such as connecting to a database, forming analytical calculations, and displaying them in various graphical and tabular formats. These can then be used in Java or JSP applications that need analytical capabilities. BI Beans use the OLAP APIs to access data.

We will now look at each of these components in detail.

9.1.3 Analytical Workspaces

Analytical workspaces allow you to store data in a multidimensional form. As with relational tables, an analytical workspace is owned by a specific schema and uses an Oracle tablespace for storage. You can use analytical workspaces to store data that is used in calculations such as forecasting and allocations.

Hint:

Existing Oracle Express databases can also be migrated into analytical workspaces in the Oracle database.

Analytical workspaces can be persistent or temporary, depending on your needs. If you need to perform a calculation but do not need to store the results, you can discard changes done within the analytical workspace at the end of the session. A temporary analytical workspace is often used for what-if analysis, where you want to try different hypothetical scenarios but not make all the changes persistent. Unlike relational tables, where changes done by DDL, such as adding a column, are automatically made visible throughout the database, all changes done within the analytical workspace are local to your session unless explicitly committed.

You can also store aggregate data within an analytical workspace, much like a materialized view; however, you will be responsible for refreshing the data. Incremental refresh mechanisms are not available for data in analytical workspaces.

Now, let us look at the physical storage model used by analytical workspaces and how it differs from the relational model.

Concepts

Throughout this book we have described how a data warehouse can be created and managed using relational tables. In this relational world, data is typically stored in a star or snowflake schema. The fact table stores various quantities that you want to analyze with respect to each dimension. Dimension objects are used to define the hierarchical relationships between various columns in the dimension tables. These dimension tables are sometimes referred to as level-based dimensions. To perform a calculation, queries must join the fact and dimension tables using appropriate predicates.

In a multidimensional format, such as an analytical workspace, there are no tables or columns. Instead, there are three main types of entities:

  • Dimension: Unlike the SQL dimension object, which expresses relationships between columns, a dimension in this world corresponds to a list of values. These are called value dimensions. For instance, a city dimension may consist of the values Boston, London, and San Francisco. Your geography dimension may consist of the values World, United States, Massachusetts, New Hampshire, UK, and London. Thus, there is no hierarchy that is implied by a dimension in itself. To specify any relationships between various values in the dimension, you must create a relation.

  • Relation: A relation stores the correspondence between a value in one dimension to another value in the same or another dimension. Note that relations can also declare a relationship between two values in the same dimension—these are called self-relations. Just as a hierarchy describes relationships in a level-based dimension, a relation describes relationships in a value-based dimension.

    You can think of the relationship defined by a relation as a parent-child dimension table in a relational schema. Figure 9.2 shows the difference between a level-based dimension table and a parent-child dimension table in the relational world.

    Click To expand
    Figure 9.2: Level-based versus parent-child dimension tables.

  • In a level-based dimension, each level is stored in a separate column. For instance, you have columns corresponding to the city, state, and region levels. On the other hand, in a parent-child dimension, all values are stored in the child column and parent column has the corresponding parent value. For example, suppose the child column contains values such as Boston, San Francisco, and MA. For each value in the child column, there will be a corresponding value in the parent column. For instance, for the child value, Boston, the parent value is MA. Unlike a level-based dimension table, where higher-level values are repeated for every lowest-level value, in a parent-child dimension every relationship is stored exactly once and hence the validity of the dimension data is automatically ensured.

  • Variable: A variable is used to store data and is equivalent to a fact table. It is defined with respect to a specific set of dimensions. Figure 9.3 shows a conceptual picture of how data is stored in a variable. In this example, the sales variable is dimensioned by geography and time. You can query the value of a variable for any values of the dimensions it is defined against. For instance, the sales value for United States for the year 2002 is $3,102. Notice that this is similar to a spreadsheet, where you can retrieve the value of any cell by simply specifying the row and column.

    Click To expand
    Figure 9.3: Conceptual view of a variable.

There are several advantages to this multidimensional storage format:

  • It enforces referential integrity. For instance, if a variable is defined along customer and time dimensions, every cell of the data will have some unique value of a customer and time. Also, relationships between dimension values are stored exactly once and hence you will not end up with inconsistent data, such as Boston, MA, and Boston, CA.

  • There is an implicit ordering among rows in the dimension that is defined at creation. This is unlike SQL, where you must explicitly add ORDER BY clauses to get data out in a certain order.

  • Users don't need to specify how to join the fact and dimension tables to get their answers. They can simply ask to report the variable for any dimension values, similar to a spreadsheet. However, unlike a spreadsheet, you are not restricted to two dimensions.

  • The data can be presented to the application as "fully solved." Once the DBA sets up the analytical workspace with various calculations, the application users do not have to describe how to perform a calculation as part of the query. They just have to indicate which of the available calculations they would like, and the calculation engine will take care of the details of computing it. The calculation may be a complex analytical function, a formula, or an aggregate. The data may be precomputed for performance or calculated on the fly; however, the application does not have to know about how it is done.

At this point, you may be wondering what is involved in creating and querying these analytical workspaces. To manipulate data stored in analytical workspaces you use the OLAP DML language. You can use OLAP DML to create dimensions and variables and to load data into the analytical workspace. However, in most situations, you probably already have your data warehouse organized in a star or snowflake schema. In this case, you can simply use the package CWM2_OLAP_AW_CREATE or the Analytic Workspace Manager Application, described later in this chapter. If you would like to use any of Oracle's tools, such as OLAP API or BI Beans, the analytical workspace must conform to a certain standard format. The package will automatically create this standard analytical workspace for you; hence, it is strongly recommended that you use these packages rather than manually create the analytical workspace elements.

9.1.4 OLAP DML

OLAP DML is a very simple but powerful language that allows you to express a variety of calculations and do spreadsheet-like reporting on data stored in an analytical workspace. It provides functions for forecasting, allocation, aggregation, statistical analysis, and financial calculations.

You can execute OLAP DML using either the OLAP Worksheet in Oracle Enterprise Manager, using the DBMS_AW PL/SQL package, or using the SPLExecutor class of the OLAP API.

To access the OLAP Worksheet from Oracle Enterprise Manager, expand the Database node in the Navigator panel, followed by the Warehouse and OLAP nodes, and then use the Object Menu. It is a simple application (similar to SQL*Plus) that allows you to execute OLAP DML commands as well as edit, compile, and run programs written in the OLAP DML language. Figure 9.4 shows the OLAP Worksheet. You issue the commands in the lower portion of the window and the results appear in the upper portion. For instance, in Figure 9.4, we are querying a dimension named OLAP_TIME_KEY.

Click To expand
Figure 9.4: OLAP worksheet in Oracle Enterprise Manager.
Hint:

OLAP DML is very different and completely separate from SQL. However, there is a SQL mode in the OLAP worksheet where you can issue regular SQL statements as in SQL*Plus. Conversely, you can issue OLAP DML commands in SQL*Plus using the DBMS_AW package.

Before you can issue any OLAP DML commands, you must first create or attach to an analytical workspace. The following AW CREATE command creates an analytical workspace named EASYDWOLAP. We are associating a specific tablespace, OLAPTBSP with it.

AW CREATE EASYDWOLAP TABLESPACE olaptbsp

In the next few sections, we will show some examples of using OLAP DML to illustrate the types of calculations that can be done with analytical workspaces. However, this is not a tutorial on OLAP DML; for that you must refer to the Oracle documentation. The OLAP worksheet also has an excellent help system, which describes all OLAP DML commands with examples.

Dimensions and Variables

Dimensions and variables are the basic elements of the multidimensional storage format. We will briefly show how they are defined, so you can understand the examples in the subsequent sections.

The first example defines an equivalent of a customer dimension with the hierarchy customer_id, town, and county.

DEFINE cust_id     DIMENSION TEXT
DEFINE cust_town   DIMENSION TEXT
DEFINE cust_county DIMENSION TEXT
DEFINE customers DIMENSION CONCAT (cust_county cust_town cust_id)
DEFINE customers.parents RELATION customers <customers>

Each level of the hierarchy cust_id, cust_town, and cust_county is specified as a dimension. The CONCAT command lets you create a single dimension—customers containing values from all three dimensions: cust_id, cust_town, cust_country. The customers.parents RELATION is a self-relation indicating the parent child relationship between various values within the customers dimension. Note that the actual relationship will be determined when we load data into these dimensions, which we will not show here.

The following group of commands defines a time dimension—very similar to the customers dimension. There are three separate dimensions—time_id, time_month, and time_year, corresponding to various levels—and a relation, timedim.parents, that establishes the parent-child relationships between the various dimension values.

DEFINE time_id    DIMENSION TEXT
DEFINE time_month DIMENSION NUMBER(2)
DEFINE time_year  DIMENSION NUMBER(4)
DEFINE timedim    DIMENSION CONCAT (time_year time_month time_id)
DEFINE timedim.parents RELATION timedim <timedim>

Next we will define a COMPOSITE dimension, which specifies what dimensions will be used together. This is useful when the data do not have all combinations of customers and timedim values; in this case a composite will only store the relevant <customers, timedim> pairs.

DEFINE dims COMPOSITE <customers timedim>

The composite dimension will be used to dimension our variables, defined in the following example. As mentioned earlier, a VARIABLE is similar to a fact table and must be dimensioned by one or more available dimensions. We will define two variables—purchase_price and ship_charge—dimensioned using the composite dimension, DIMS, defined earlier.

DEFINE purchase_price
       VARIABLE NUMBER (6,2) <dims <customers timedim>>

DEFINE ship_charge
       VARIABLE NUMBER (6,2) <dims <customers timedim>>

We will use these dimensions and variables in the examples in the following sections.

Reporting data with OLAP DML

We can report the data in the variable along its dimensions using the REPORT command. The REPORT command returns the answer, similar to a spreadsheet, with one dimension along the rows and another along the columns. You can restrict the data you are querying to a specific value or list of values using the LIMIT command. This is similar to a selection specified by a WHERE clause in SQL. However, unlike SQL, where the selection is specified on a query basis, the LIMIT commands in an OLAP DML persist as long as you are attached to the analytical workspace.

In the following example, we are querying the purchase_price variable for the cust_county dimension for months January through March. The keyword ACROSS indicates that the time_month dimension values will be reported as columns. Notice how the engine automatically figured out the dimensions involved, using the definition of the variable.

LIMIT customers to cust_county
LIMIT time_month to 1 to 3

REPORT across time_month : purchase_price

                                 ---------PURCHASE_PRICE---------
                                 -----------TIME_MONTH-----------
CUSTOMERS                            1          2          3
-------------------------------- ---------- ---------- ----------
<CUST_COUNTY: Hants>               1,976.03   1,924.90   3,286.77
<CUST_COUNTY: London>              2,335.90   2,555.28   3,122.70
<CUST_COUNTY: Mersey>              2,335.90   2,555.28   3,122.70

You can also compute other variables using the variables defined earlier. For instance, the following example defines a variable named totalsales that computes the sum of purchase_price and shipping charge.

DEFINE TOTALSALES VARIABLE NUMBER (6,2)  <DIMS <CUSTOMERS TIMEDIM>>

We then compute this variable for the lowest levels of the customers and timedim dimensions.

LIMIT timedim TO time_id
LIMIT customers TO cust_id
ACROSS CUSTOMERS TIMEDIM
     DO 'TOTALSALES = PURCHASE_PRICE + SHIP_CHARGE'

We can now report on this new variable. In the following example, we are asking for the totalsales for the first three months for each customer. Notice that we did not have to specify how to compute totalsales as part of the query. This is where the power of OLAP DML becomes apparent.

LIMIT CUSTOMERS TO CUST_ID
LIMIT TIME_MONTH TO 1 TO 3

REPORT across time_month : totalsales

                        -----------TOTALSALES-----------

                        -----------TIME_MONTH-----------
CUSTOMERS                     1          2          3
-------------           ---------- ---------- ----------
<CUST_ID: AB123456>          32.51     134.24     143.46
<CUST_ID: AB123457>          65.02     188.42      65.02
...

Aggregating data

You can use OLAP DML to perform aggregation. Let us assume that we have loaded the data corresponding to the cust_id and time_id values into the variable purchase_price. We would now like to compute the values for the higher levels, such as cust_county and time_month, using aggregation. To aggregate the data, we must define an aggregation map using the AGGMAP command. The aggregation map gives the calculation engine instructions on how the aggregation is to be performed—which aggregate function to apply, which levels to aggregate over, and what aggregates to precompute. Once the DBA has defined the aggregation map, it can be used without knowing the details of how the aggregation is performed underneath.

The following AGGMAP indicates that you want to compute aggregates over the time_id dimension and over the cust_id dimension. The AGGMAP specifies the relations that determine how to aggregate from one level to the next.

DEFINE easyagg AGGMAP
AGGMAP
  RELATION timedim.parents PRECOMPUTE (ALL)
  RELATION customers.parents PRECOMPUTE (ALL)
END

You can now report the data using the previous AGGMAP, as follows:

>report across time_month : aggregate(ship_charge using easyagg)

                                 ----AGGREGATE(SHIP_CHARGE-------
                                 --------USING EASYAGG1)---------
                                 -----------TIME_MONTH-----------
CUSTOMERS                            1          2          3
-------------------------------- ---------- ---------- ----------
<CUST_TOWN: Eastleigh>                 2.95       7.45      11.95
<CUST_TOWN: Soton>                    10.40      14.90      23.90
<CUST_TOWN: Chandlers>                 2.95       5.90      11.95
<CUST_TOWN: Soton2>                    2.95       5.90      11.95
<CUST_TOWN: Soton3>                    2.95       5.90       2.95
...

In the above example, we asked to precompute all aggregates but we could get quite sophisticated and ask to precompute only some levels or only some values.

Forecasting

One of the common operations performed using OLAP DML is forecasting. This is one of the many advanced analytical operations that can be performed with analytical workspaces and OLAP DML but not in SQL. To forecast a quantity such as sales we must perform the following steps:

  • Define variables to store the forecast results.

  • Specify the parameters of the forecast.

  • Execute the forecast.

We will show a very simple example of forecasting sales for the next 6 months based on the past 6 months.

The first step is to define a variable called SALES_FORECAST, which stores the result of the forecast. Note again that we have dimensioned this variable using the composite dimension DIMS.

DEFINE SALES_FORECAST VARIABLE NUMBER(6,2) <DIMS <TIMEDIM CUSTDIM>>

Next, we constrain timedim to the time_month level and customers to the cust_id level. This means that the forecast will be computed using the months in the time dimension, for each cust_id value in the customers dimension.

LIMIT TIMEDIM TO TIME_MONTH
LIMIT CUSTOMERS TO CUST_ID

To specify parameters and run the forecast, we must create a handle, which will be used by subsequent commands. The handle, called sf_handle, is obtained by calling the FCOPEN command.

DEFINE sf_handle VARIABLE INTEGER;

sf_handle = FCOPEN('EasyDWSalesForecast')

Next, we will set the forecast parameters using the FCSET command. We are using the automatic method for forecasting. We will consider 6 time periods (months) as historical data and forecast using a periodicity parameter of 2.

FCSET sf_handle method 'automatic' histperiods 6 periodicity 2

Finally, we execute the forecast using the FCEXEC command. We must specify the name of the time dimension and also the variable containing the data to be used for the forecast—in our case, purchase_price. The results are placed into the sales_forecast variable defined earlier.

FCEXEC sf_handle TIME timedim INTO sales_forecast purchase_price

Finally, we close the handle, as follows:

FCCLOSE sf_handle

You can now use an AGGMAP, discussed earlier, to aggregate the sales forecast to higher dimension levels for reporting purposes. Assuming we have done so, we can now report the sales_forecast variable as before to see the forecasted results for the months 9 through 11.

LIMIT time_month to 9 to 11
LIMIT customers to cust_id

REPORT across time_month: sales_forecast

                        ---------SALES_FORECAST---------
                        -----------TIME_MONTH-----------
CUSTOMERS                     9          10          11
-------------           ---------- ---------- ----------
<CUST_ID: AB123456>         232.51     334.24     343.46
<CUST_ID: AB123457>         165.02     188.42     265.02
...

Creating custom measures

With OLAP DML you can also define formulae to perform calculations using simple arithmetic or analytical functions. Once defined, users can then reference these calculations in reports like any other variables defined earlier. Again, we should emphasize that the end user does not need to know how the calculation was done or what analytical function it used.

The following example defines a simple FORMULA for totalsales as a total of purchase_price and ship_charge.

DEFINE TOTALSALES FORMULA DECIMAL <TIMEDIM CUSTOMERS>
EQ purchase_price + ship_charge

The following example reports the totalsales values for months 4, 5, and 6.

LIMIT time_month TO 4 to 6
LIMIT timedim TO time_month

REPORT across cust_dim : totalsales

                -------------------TOTALSALES-------------------
                ------------------CUST_COUNTY------------------
TIMEDIM              Hants          LondonC         Mersey
--------------- --------------- --------------- ---------------
<TIME_MONTH: 4>        6,376.96        7,687.26        7,687.26
<TIME_MONTH: 5>        4,493.81        5,363.68        5,363.68
<TIME_MONTH: 6>       13,749.80       16,663.60       16,663.60

The next example defines a more complex formula to compute change in sales from one time period to the prior time period using the LAGDIF function. The NOSTATUS keyword indicates that we will compute the LAGDIF using all dimension values for the timedim dimension, regardless of those selected by the current LIMIT specification

DEFINE SALES_PREV_MONTH FORMULA
DECIMAL <TIMEDIM CUSTOMERS>
EQ LAGDIF(PURCHASE_PRICE,1,TIMEDIM,NOSTATUS)

We will report it with the same limit settings as the previous example.

REPORT across cust_dim: sales_prev_month

                ----------------SALES_PREV_MONTH---------------
                ------------------CUST_COUNTY------------------
TIMEDIM              Hants          LondonC         Mersey
--------------- --------------- --------------- ---------------
<TIME_MONTH: 4>        3,885.18        4,481.96        4,481.96
<TIME_MONTH: 5>       -1,883.15       -2,323.58       -2,323.58
<TIME_MONTH: 6>        9,254.98       11,298.91       11,298.91

In this section, we have given you a quick but broad overview of OLAP DML and analytical workspaces. We saw how data is stored, aggregated, calculated, and reported in the multidimensional format. We have only scratched the surface of what can be done with OLAP DML, but hopefully you have some idea of the simplicity and power of this language.

Besides interactively issuing OLAP DML using the OLAP Worksheet, you can also use it within an application, as discussed in the next section.

9.1.5 Accessing analytical workspaces in an application

Oracle OLAP provides several ways by which an application can access multidimensional data in an analytical workspace. These are summarized in Figure 9.5. A PL/SQL application using OLAP DML can access an analytical workspace using the DBMS_AW PL/SQL package. Java programs can access it using the OLAP API, provided that appropriate OLAP catalog metadata has been defined. SQL applications can also access the data by defining table functions, which convert the data into rows. If necessary, relational views may be defined on top of the table functions, so access to the analytical workspace is further encapsulated.

Click To expand
Figure 9.5: Application access to analytical workspaces.

We will now briefly look at each of these mechanisms.

DBMS_AW package

The DBMS_AW package provides functions that allow you to execute OLAP DML commands and programs using PL/SQL programs or SQL*Plus.

The EXECUTE procedure can be used to execute one or more OLAP DML commands and print the output to the screen using the DBMS_OUTPUT package. The following example attaches to the EASYD-WOLAP analytical workspace and reports the cust_town dimension.

SET SERVEROUTPUT ON;
BEGIN
  DBMS_AW.EXECUTE('aw attach easydwolap
                   report cust_town');
END;
/
CUST_TOWN
--------------
Eastleigh
Soton
Chandlers
Soton2
Soton3
Soton4
Soton5
...

SQL access to multidimensional data

Applications can access multidimensional data stored in analytical workspaces with SQL by using SQL table functions. Oracle provides a table function called OLAP_TABLE to do this but you can also write your own custom table functions. For instance, suppose we want to retrieve the data from the analytical workspace EASYDWOLAP defined earlier. We would like to get the purchase_price along the customer and time dimensions. To use the OLAP_TABLE function, as with any other table function described in Chapter 5, you must first define a TYPE that describes the rows being returned and a TYPE that describes a TABLE whose rows are objects of this type.

For our example, we define a type purchase_price_type as follows:

CREATE TYPE purchase_price_type AS OBJECT
(cust_id VARCHAR2(80),
 time_id VARCHAR2(80),
 purchase_price NUMBER);
/

Next, we define a purchase_price_type, which describes a table whose rows are of the purchase_price_type.

CREATE TYPE purchase_price_tbl AS TABLE OF purchase_price_type;
/

We can then get the purchase_price along the customer and time dimensions using the OLAP_TABLE function, as follows:

SELECT * FROM TABLE(OLAP_TABLE(
'easydwolap duration session', 'purchase_price_tbl', '',
'measure purchase_price from purchase_price
 dimension cust_id from cust_id
 dimension time_id from time_id'));

The OLAP_TABLE function takes as parameters the name of an analytical workspace and its duration (easydwolap duration session), the type of the result (purchase_price_tbl) and OLAP DML commands to retrieve the various columns for the result. It also takes an optional limit parameter, which allows you to retrieve only partial data.

You can now create a relational view on this table function, as follows, so that applications can access this data without really needing to know about table functions and OLAP DML.

CREATE VIEW purchase_price_vw
as
SELECT * FROM TABLE(OLAP_TABLE(
'easydwolap duration session', 'purchase_price_tbl', '',
'measure purchase_price from purchase_price
 dimension cust_id from cust_id
 dimension time_id from time_id'));

This view is now queried like any other relational table. For instance, in the following example, we are aggregating the purchase_price using SUM.

SELECT time_id, SUM(purchase_price) as purchases
FROM purchase_price_vw
GROUP BY time_id;

TIME_ID      SUM(PURCHASE_PRICE)
--------------------------------
01JAN2002    19859.46
01FEB2002    19188.96
01APR2002    44660.79
...

This was a very simple example, and you can get quite sophisticated with your use of table functions. With this feature, the application can now be completely unaware of whether the data being accessed is stored in a relational or a multidimensional format.

The OLAP API

The OLAP API is a set of Java classes that can be used to develop OLAP applications. It is very well suited to developing thin-client applications, which can be accessed with a Web browser. The OLAP API uses a multidimensional model for querying data; however, it internally translates these queries into SQL. To access objects using this API, you need to have defined the objects in the OLAP catalog, described in the next section. The underlying data may be stored either in relational tables or in analytical workspaces encapsulated within relational views.

A detailed discussion of the OLAP API is beyond the scope of this book. Instead, we will illustrate BI Beans, which are reusable components that use the OLAP API to access data. These can then be incorporated into custom OLAP applications.

In the next section, we will discuss the OLAP catalog, which defines metadata required by tools such as OLAP API, BI Beans, and the Analytic Workspace Manager Application.

9.1.6 The OLAP catalog

The OLAP catalog stores the metadata to specify the logical model of your data. If your application wants to have access to data using OLAP API, then it must be defined in the OLAP catalog.

The OLAP metadata consists of the following types of entities:

  • Dimensions: Dimensions are used to express relationships such as hierarchies in your data. Dimensions consist of levels, hierarchies, and level attributes. In addition, you can also specify some dimension attributes, such as its description.

  • Measures: A measure is a quantity that will be used in calculations, such as purchase price or cost.

  • Measure folders: A measure folder, also known as a catalog, is a convenient place to keep related measures together.

  • Cube: A cube defines how measures will be aggregated across one or more dimensions. In relational terms, it defines how to join your fact and dimension tables. A cube also specifies which hierarchies in the dimensions will be used to compute aggregations.

The OLAP catalog requires you to map these elements to relational tables and columns. Hence, if you have data in analytical workspaces, you must encapsulate access to the data within relational views, as described previously.

OLAP metadata can be defined in two ways:

  • Using the CWM and CWM2 packages

  • Using Oracle Enterprise Manager

The CWM packages correspond to the first version of the Common Warehouse Metadata, also known as CWMLite. It supports traditional dimensions, as defined in a star or a snowflake schema. A dimension defined using CWM API in the OLAP catalog has a SQL dimension object (described in Chapter 4) underlying it, in addition to OLAP metadata. Data warehouse design tools such as Oracle Warehouse Builder also allow you to automatically generate metadata according to the CWM specification.

CWM2 is the second version of CWM and provides advanced features, such as parent-child dimensions, ragged hierarchies (where not all data has the same lowest level in the hierarchy), same value mapping to different levels in different hierarchies, and NULL values in level columns.

At the time of writing, Oracle Enterprise Manager only provides a graphical interface to create the CWM version of metadata. If your data is stored in relational tables and follows a star or a snowflake schema, this should be sufficient for you.

We will now illustrate the use of Oracle Enterprise Manager to generate OLAP metadata. The CWM APIs are very easy to use and follow the same flow as the wizards in Enterprise Manager and hence we will not discuss them in this book.

Creating OLAP metadata in Oracle Enterprise Manager

To access the OLAP functionality in Oracle Enterprise Manager, expand the Database node, followed by the Warehouse node, and then the OLAP node. Oracle Enterprise Manager provides wizards to create a dimension and a cube. It also allows you to create measure folders.

If you have any SQL dimension objects defined for use with query rewrite (described in Chapter 4), you will automatically see them listed under Dimensions. If you click on one of them, you may get a window saying that some of the metadata needed for OLAP has not been created yet. At this point, you can ask it to continue. It will generate several CWM API calls, as shown in Figure 9.6. Be assured that your existing dimension object will not be harmed in any way by doing this. Once you press Apply, the CWM metadata will be created. You can now use this dimension to define your cube objects as discussed next.

Click To expand
Figure 9.6: Dimensions for OLAP in Oracle Enterprise Manager.
Hint:

Under the OLAP options tab in Figure 9.6, you can set up descriptive names for dimensions, levels, and hierarchies. It is recommended that you provide all this information if you plan to use the BI Beans Wizards.

Oracle Enterprise Manager allows you to define cube metadata that relates your dimensions and measures. We will walk through some of the steps performed to create a cube, using the Cube Wizard. You can access the wizard in the Object menu in Figure 9.6.

First, you will be asked to name the cube and also select the schema where it should be placed (not shown here). Next, you will be asked to choose the fact table that contains the measures to be stored in the cube, as shown in Figure 9.7. In this example, we have chosen EASYDW.PURCHASES table.

Click To expand
Figure 9.7: Cube Wizard—choosing fact tables.

After this, you will be asked to pick the dimensions you would like to include into your cube, as shown in Figure 9.8. In our example, we have two dimensions, CUSTOMER_DIM and OLAP_TIME_DIM, which we assume were created earlier.

Click To expand
Figure 9.8: Cube Wizard—choosing dimensions.

In the next screen, you will be asked to define how your dimensions join to your fact table. You must define this relationship for each dimension in the cube before you will be allowed to move on. You must also define a default hierarchy along which the cube will compute the measures. In Figure 9.9, we have specified that the CUSTOMER level in the CUSTOMER_DIM dimension joins to the fact table using the customer_id column.

Click To expand
Figure 9.9: Cube Wizard—joining dimension and fact tables.

The next step, shown in Figure 9.10, is to define the measures to be stored in the cube. You must name each measure and indicate what fact table column it corresponds to. By default any column in the fact table with a data type of NUMBER will be included in the measure list. Note that the default aggregation method used for each measure is SUM.

Click To expand
Figure 9.10: Cube Wizard—specifying measures.

Finally, you will be shown a summary of the cube, as shown in Figure 9.11. You can press the Show SQL button to see the CWM APIs that will be used to generate the cube.

Click To expand
Figure 9.11: Cube Wizard—summary screen.

When you press Finish in Figure 9.11, the cube will be created. If you chose to invoke the OLAP Summary Advisor by checking the box at the bottom of Figure 9.11, it will then be invoked. The OLAP Summary Advisor uses the Summary Advisor described in Chapter 4 in a special way to generate materialized views to optimize the cube. However, these materialized views have some additional OLAP metadata associated with them, so these must not be confused with materialized views used for other applications. You must use the appropriate Summary Advisor for your application.

Hint:

The OLAP Summary Advisor is available when you right-click on a cube object in Oracle Enterprise Manager. The Summary Advisor, described in Chapter 4, is available by expanding the Summary Management node.

Once the cube has been created, you can edit any of the cube's properties by right-clicking on that cube under the OLAP Management section. You can also browse the data in the cube using the Cube Viewer application in Oracle Enterprise Manager. If you select the cube you want to view and right-click on it—you will find the Cube Viewer in the pop-up menu. The Cube Viewer will display the results of any query in a cross-tabulation format, as shown in Figure 9.12. In this example, we are displaying the purchase_price measure for all products in Hants county. You can drill down into the data further by clicking on the arrows in the title cells.

Click To expand
Figure 9.12: Cube Viewer.

The query whose results are to be displayed can be constructed or modified using the Query Builder from the File menu. The Query Builder is very similar to the one used by BI Beans, discussed later; hence, we will not delve into its details here. However, unlike BI Beans, where you can customize this report in a variety of ways, the Cube Viewer is just a simple application to peruse your data and is a convenient way to verify that the OLAP metadata has been defined correctly.

9.1.7 Analytical workspace creation tools

A situation likely to be encountered by many users is that they already have a relational warehouse and would like to build an analytical workspace for some of its special features, such as forecasting and what-if analysis. Oracle provides two simple tools to easily create an analytical workspace from a star or snowflake schema. These tools will ensure that the analytical workspace is created in such a way that it can be used by all of Oracle's analysis tools, such as OLAP API and BI Beans.

In this section, we will briefly describe the two tools provided for this purpose:

  • CWM2_OLAP_AW_CREATE, a PL/SQL package

  • Analytic Workspace Manager Application

To create an analytical workspace using either of these tools, your data must be in a star or a snowflake schema. Also, you must have defined CUBE and DIMENSION metadata in the OLAP catalog.

The CWM2_OLAP_AW_CREATE package allows you to:

  • Define dimensions in the analytical workspace, corresponding to a dimension in the OLAP Metadata, using the AW_DIMENSION_CREATE procedure.

  • Define variables, corresponding to a cube in the OLAP metadata, using the AW_CUBE_CREATE procedure.

  • Load data into the dimension and variables in the analytical workspace from the fact and dimension tables, using various load procedures.

  • Create relational views on top of the analytical workspace using the AW_DIMENSION_CREATE_ACCESS and AW_CUBE_CREATE_ACCESS procedures. This allows the analytical workspace to be accessed via SQL.

Oracle also provides a tool known as the Analytic Workspace Manager, which allows you to do the same using a graphical interface. We will demonstrate some of its features next.

Hint:

The Analytic Workspace Manager Application is available free of charge at Oracle Technology Network (http://otn.oracle.com). It is not part of the Oracle Database installation CD.

The Analytic Workspace Manager Application allows you to create and edit dimensions and variables in an analytical workspace. For instance, in Figure 9.13, we are editing a variable named COST.

Click To expand
Figure 9.13: Analytic Workspace Manager.

The Analytic Workspace Manager also allows you to browse the cubes and dimensions in the OLAP catalog. Also included is a wizard to create an analytical workspace from a cube defined in the OLAP catalog. Figure 9.14 shows the first screen of the wizard, where you select the cube you would like. In Figure 9.14, we have chosen the PURCHASES cube assumed to have been defined earlier.

Click To expand
Figure 9.14: Analytic Workspace Creation Wizard— Select Cubes.

Next, you will be asked to name the analytical workspace and choose a tablespace to store it, as shown in Figure 9.15.

Click To expand
Figure 9.15: Analytic Workspace Creation Wizard— Select Analytic Workspace.

Finally, if you would like to enable the use of this analytical workspace with OLAP API and BI Beans, you can check the option in Figure 9.16. This will generate various scripts to create relational views on top the analytical workspace elements. You can specify the directory where the scripts should be stored and run them at your convenience.

Click To expand
Figure 9.16: Analytic Workspace Creation Wizard— Enable Workspace for OLAP API & BI Beans.

It is highly recommended that you make use of these tools to create analytical workspaces rather than creating them manually, which can be quite cumbersome.

9.1.8 BI Beans

Business Intelligence Beans (BI Beans) is a set of reusable components that allows you to rapidly develop OLAP applications. They are integrated into Oracle's JDeveloper product, where they can be created and customized using simple wizards. BI Beans can perform a wide range of tasks, such as connecting to a database, building queries to perform analytical calculations, and displaying the results in extremely powerful reports, tables, or graphs. These components can then be easily deployed as part of a Java or JSP application.

Note

To use BI Beans with 9i Release 2 OLAP Option, you must use JDeveloper 9.0.3 and BI Beans 2.7 or later.

We will illustrate the capabilities of BI Beans in the following text. In these examples, we will assume that we have defined a cube called PURCHASE_DATA, which has two dimensions, PRODUCT and CUSTOMERS, and two measures purchase_price and shipping_charge, which are grouped into a measure folder called Purchase Measures. The descriptive attributes supplied when defining dimensions and cubes will be used to display various measures and dimensions in these wizards.

A novel aspect of BI Beans is that you have full access to the data as the component is being designed; so you can immediately see how the resulting graph or presentation will look. Before you start to build business intelligence components, you must set up a database connection using the Designer bean. Then you can use the Query Builder to define queries, Presentation Wizard to create a report or graph, and Calculation Builder to define calculations.

Creating reports

The Presentation Wizard, shown in Figure 9.17, allows you to create reports in a Crosstab, Graph, or Table format. We will first create a report of total sales in Hants county for various product categories. As we do different customizations with this report, you should begin to appreciate the power of BI Beans. In Figure 9.17, we have chosen to create a crosstab named EasyDWSalesAnalysis.

Click To expand
Figure 9.17: New Presentation Wizard.

The next step, shown in Figure 9.18 is to choose which measures we want in the report. The Automatically add/remove dimensions box is checked, and, hence, the dimensions associated with the measure will be automatically included in the report. Note that some of the screens used by the Presentation Wizard are also common to the Query Builder Wizard, since ultimately a presentation is just a visual representation of a query.

Click To expand
Figure 9.18: New Presentation Wizard—choosing measures and dimensions.

The next step is to select how these dimensions will be laid out in the report. In a crosstab format, you can represent data along two dimensions—a row dimension and a column dimension and, optionally, a third page dimension. The wizard will choose a default layout for the dimensions. You can change the layout by a simple drag-and-drop operation to move the dimensions into the row, column, or page as desired. In Figure 9.19, since we have only two dimensions, we will keep customers as the columns and products along the rows and will not use the page dimension.

Click To expand
Figure 9.19: Layout of a crosstab.

We then choose what levels or values of the dimension to include in the report, as shown in Figure 9.20. In our example, we have chosen only Hants county. Note that even though this is a design environment, the wizard shows you the actual values in your customer dimension.

Click To expand
Figure 9.20: Choosing dimension members.

On the right-hand side in Figure 9.20, is the Selected panel with two tabs—Steps and Members. The Members tab shows the values chosen for a particular dimension. The Steps tab shows how you determine the list of members. In our example, we only have one member, Hants county. Later, we will discuss how you can add members according to specific conditions.

We could continue on and customize the product dimension, but for now let's just press the Finish button and customize it later. The resulting report will look like that shown in Figure 9.21. You may wonder what is so special about this report. Well, you will notice that there are small arrows in the title cells for ALL_PRODUCTS, ELEC, HDRW, and MUSC. If we clicked on ELEC to expand it, the bean would automatically query the data to drill down to the individual product level. In fact, we have expanded the ALL_PRODUCTS cell to drill down to the individual categories. If you include this into a JSP application, the users now have an interactive report where they can roll up and drill down the hierarchy. Along the customer dimension, we have chosen Hants county as the top level, which means we can drill down to towns and individual customers in this county.

Click To expand
Figure 9.21: The EASYDW sales analysis report.

Let's now see how we can edit and customize this report. In Figure 9.21, there is a toolbar just above the window where the report is displayed. If we click on the first button in this toolbar (with the tooltip Show Query Builder), we get the Query Wizard, shown in Figure 9.22, where we can edit all aspects of the report.

Click To expand
Figure 9.22: Customizing the content with Query Wizard—Conditions.

Earlier we mentioned Steps and Members; let us now discuss the Conditions tab. Each dimension in the report can be customized to include members according to a certain condition. The wizard will automatically show you some useful conditions pertinent to the data. In Figure 9.22, we have chosen "Top 10 based on Purchase Price." The steps indicate that we will start with ALL_PRODUCTS and then add the top ten products. So the report will now consist of data for ALL_PRODUCTS (from which we can drill down) and also the top ten products, as shown in Figure 9.23.

Click To expand
Figure 9.23: EASYDW analysis report with top ten products.

Data-driven formatting

As with all reporting tools, you can customize this report to add titles, change fonts, and set colors. The Customizer, shown in Figure 9.24, is available by clicking the button next to the Query Wizard in the toolbar above the report. We have added a title and a subtitle to our report.

Click To expand
Figure 9.24: Customizing the crosstab look and feel.

Another interesting aspect of BI Beans is that you can actually format the report using the actual data. For instance, we would like the report to highlight products that are not performing well. For this, we will go to the Format tab and click the Data button to get the window shown in Figure 9.25. We will create a format named "Losers," which sets the background color Gray for all cells in the report where the total purchase_price is less

Click To expand
Figure 9.25: Data-driven formatting.

than 8,000. To do this we go to the Rules tab, shown in Figure 9.26, and edit the data to add the required conditions. For instance, we are setting the condition purchase_price < 8,000, as shown in Figure 9.27.

Click To expand
Figure 9.26: Specifying data-driven formatting rules.

Figure 9.27: Specifying a condition on purchase price.

The resulting report is shown in Figure 9.28. Hopefully, you now have an appreciation of how easy it is to create very nice and powerful reports. Even though we are showing these reports in operation within JDeveloper, you are seeing these reports exactly as they would appear if included in your application.

Click To expand
Figure 9.28: EASYDW analysis report with badly performing products highlighted.

Graphs

The Presentation Wizard can also be used to create graphs. The steps used to pick dimensions and measures are identical to those for the crosstab. Figure 9.29 shows the various types of graphs available.

Click To expand
Figure 9.29: Graphs.

As with crosstabs, a graph can display data along multiple dimensions, as shown in Figure 9.30. Instead of rows and columns, in a bar graph you have bars representing one dimension and groups representing a second dimension. You can also have a page dimension as a third dimension.

Click To expand
Figure 9.30: Specifying the layout of a graph.

Figure 9.31 shows a graph created using the presentation bean, where we see sales by customer county. In this example, we have chosen customer counties as groups and products as bars.

Click To expand
Figure 9.31: Graph of product sales by county.

Note that this graph also has drill down capability—if you click on the bars, it will drill down to the next level, which is the product category, as shown in Figure 9.32. Now you can see the most popular product category in each county. You can also customize the graph using the data—for instance, we have chosen specific colors for each category value.

Click To expand
Figure 9.32: Drilling down in the graph.

Calculations

The examples so far have produced reports from the base data; however, most OLAP applications will need to perform calculations. These calculations may involve analytical functions, such as ranking, reporting aggregates, cumulative sums, and moving averages. The Calculation Builder allows you to specify these calculations using measures in your cube. In Figure 9.33, we are creating a share calculation, which computes the ratio of a quantity with respect to the total (recall the RATIO_TO_REPORT function discussed in Chapter 3).

Click To expand
Figure 9.33: Creating a calculation.

On pressing the Next button, you get the screen shown in Figure 9.34, where you specify the parameters for the calculation. In this example, we have specified the measure purchase price and indicated that the ratio would be computed for each level in the products hierarchy with respect to the total. So we will be able to find out the market share of sales for each product category.

Click To expand
Figure 9.34: Specifying the parameters of the calculation.

These calculations will then be available to be included in presentations such as graphs or reports. For instance, in Figure 9.35, we have plotted a pie graph of sales for each category for each county.

Click To expand
Figure 9.35: Graphing a Calculation

9.1.9 Summary

In this section, we have given you an overview of the components and capabilities of Oracle 9i OLAP. Regardless of whether you store your data in relational or multidimensional format, the Oracle database provides all the tools and techniques required for advanced business analysis applications.


Oracle Data Mining

9.2 Oracle Data Mining

Data mining is a process that finds hidden patterns and relationships in data to help you make better business decisions.

In the course of running a business, you may encounter data in different forms. Operational data tells you day-to-day information about your customers—what did John Doe buy on August 5, 2002? In the data warehouse, data is organized according to predefined relationships using fact and dimension tables. You can then use OLAP tools to find out various facts about your business. How many people bought electronic products from your store and what was the average amount they spent? How many customers who bought electronic products were engineers? These are all questions based on existing data.

However, you may also like answers to questions such as:

  • Which of your customers are likely to buy a personal video recorder?

  • What could be a good city to launch a new product?

  • What might be a good promotion to offer for Christmas?

This is where you need some predictive insight into your data. Data mining uses various statistical and machine learning techniques to discover trends and patterns of behavior. In some cases, you may know what you are looking for. In other cases, there is no clear definition of what the mining process may find. You are saying: Here is my data—go find me something interesting to look at. This is indeed the challenge of data mining—it is like finding a needle in a haystack, only you may not know what the needle looks like! One of the biggest advantages of building a data warehouse is that data has been cleansed and consolidated and is now more amenable to data mining.

Data mining can be used in many applications, such as improving store layouts, fraud detection, mail-order promotions, reducing customer churn, and Web site personalization. It can also be used in life-sciences applications, such as finding patients at high risk for certain diseases.

Oracle 9i data mining option

Oracle data mining is an option to Oracle 9i Enterprise Edition that embeds data mining functionality in the server. Data mining involves processing large amounts of data. The data often needs to be preprocessed and put into a format suitable for the mining algorithms. By incorporating mining functionality into the database, the mining algorithms can take advantage of Oracle's scalability and features such as partitioning, compression, and parallel execution to speed up the analysis.

Oracle data mining is a Java API that supports association rules, classification, and clustering. Oracle is the lead sponsor of the Java Data Mining (JSR-73) Specification for data mining. Over future releases, Oracle data mining will conform to this standard once Java data mining attains reference status.

Oracle data mining should not be confused with Darwin, also known as Oracle Data Mining Suite. This product did the data mining analysis outside the database and has now been discontinued. Darwin had a graphical interface with point-and-click capability, but it was extremely difficult, if not impossible, to integrate data mining directly into your applications with this product. On the other hand, Oracle data mining allows you to embed data mining directly into your application logic and workflow, rather than extracting data out of your applications for analysis.

Unlike much of the other material in this book, data mining algorithms need some mathematical background, such as statistics and probability. In this section, we will provide you with a broad overview of the data mining process and techniques while avoiding the technical details of each algorithm. We will describe the tasks and briefly illustrate the use of Oracle Data Mining Java API. To read the examples at the end of this chapter, you must have a basic knowledge of the Java programming language. However, if you are only looking for a conceptual overview, you may safely skip these examples.

9.2.1 Oracle data mining techniques

Data mining problems can be classified into two categories. In some situations you have some idea of what you are looking for—for example, you are interested in customers who are likely to buy a digital camera. This is known as directed or supervised learning. In other cases, you leave it to the mining process to find you something interesting—for example, a high incidence of accidents at a certain intersection. This is called undirected or unsupervised learning.

Oracle data mining supports various techniques for mining data, each of which falls into one of these two categories.

  • Association (undirected)

  • Clustering (undirected)

  • Classification (directed)

We will look at each of these in some detail.

Association rules

You are probably familiar with Amazon.com's feature: Hello Jane! We have some new recommendations for you. This type of Web site personalization uses a data mining technique known as association rules. Mining with association rules finds items that occur together frequently. For instance, it may find that a large percentage of users who bought the book The Lord of the Rings also bought the book The Hobbit. So if you buy The Lord of the Rings, it may recommend you also read The Hobbit.

Data mining using association rules is also known as market-basket analysis. When you visit your local grocery store, you may find that the seafood department has lemons or tartar sauce next to the fish. This is because they may have found that 80 percent of people who buy fish also buy lemons to go with it. Most of these stores offer some kind of frequent shopper cards—by keeping a count of what combinations of items have been bought by the same person, they can organize their shelves more appropriately and even send you coupons for the same or similar items for your next visit.

Mining with association rules involves counting how many times a certain group of items occur together. In this case, you do not necessarily know what combination to look for—hence, this comes under the unsupervised or undirected learning category. The association algorithm comes up with rules of the type "A implies B." There are two quantities of interest to the user of this algorithm—support and confidence.

  • Support tells us the percentage of the transactions where the combination of items A and B occur together. It helps identify combinations that are sufficiently frequent (e.g., purchasing fish or purchasing fish and lemons).

  • Confidence tells us what percent of transactions that have an item A also have an item B (e.g., how many transactions that have fish also have lemons).

To use association rules, a user must provide the desired level of support and confidence for a rule to be considered interesting. For instance, suppose we specify a support of 20 percent and confidence of 70 percent. If we found that 50 percent of all transactions involve fish and 40 percent of all transactions involved the combination fish and lemons, then the combination "fish, lemons" is interesting because it exceeds the minimum support of 20 percent. In this example, the combination "fish, lemons" has confidence (40/50) * 100 = 80%, so it also meets the minimum confidence criterion. Hence, "fish implies lemon" will be reported as an association rule.

To make meaningful business decisions, both support and confidence are important. Consider this alternate example of an area where fish is not very popular due to high levels of mercury. Maybe only 5 percent of the transactions involve fish. The item "fish" does not have enough support, and hence the rule "fish implies lemons" will not make any significant difference to our sales.

Figure 9.36 shows a typical output of mining analysis using association rules, as analyzed using Oracle 9i Discoverer. The first two rules say people often buy wine or lemons to go with fish.

Click To expand
Figure 9.36: Association rules.

Clustering

Clustering is a technique used to divide a large data set with many attributes into a small number of closely packed groups. Such groups are not easily apparent to a human eye due to the large number of attributes involved. For example, suppose you had census data for a population, including several attributes such as age, occupation, occurrence of diseases, and so on. By clustering this data, you may find that there are several pockets where a certain disease is prevalent, possibly pointing to a polluted water supply in those regions. Since we have no definite idea of what we may find, this is another example of unsupervised learning.

The groups generated by a good clustering algorithm are such that all items in one group are quite alike each other in some respect and very much unlike items in other groups. Figure 9.37 illustrates the concept of clusters in data. In this figure, two attributes of the data has been plotted using a scatter plot to highlight the clusters. For all items in each cluster, the values of the two attributes have greater similarity compared with items in different clusters. This is an example where the clusters in the data were apparent by simply plotting a graph. In practice, it is not often possible to visualize clusters in this way, since we may have more than two or three dimensions! For instance, in bioinformatics applications, you can have thousands of dimensions! In these situations, we use mathematical clustering algorithms to identify clusters.

Click To expand
Figure 9.37: Clustering.

One of the applications where clustering is used is market segmentation. Suppose you were a large retailer selling a wide range of products from soaps to jukeboxes. With clustering, you can segment your customer base into groups based on demographics or past buying habits. This allows you to customize your advertising strategy for each segment and to better serve the more profitable segments.

Clustering is performed by first analyzing a small section of the data to determine clusters. Once the clusters have been determined, the remaining data is then analyzed to assign each individual item to a cluster, with a certain probability.

Classification

Suppose you wanted to target a promotion for a new digital camera and would like to know which of your customers are likely to buy the camera. Classification is a data mining technique that is useful for this application. Classification divides data into two or more well-defined classes. Unlike clustering, where you do not know what groups will be generated, in classification you know exactly what each group represents. In the previous example, the two groups are customers who are likely to buy a camera and customers who are not likely to buy the camera. This is an example of supervised learning.

In classification, you first analyze a small part of your data to build a model. For instance, you would analyze real data for people who have bought digital cameras over a given time period. The data used to build a model is known as build data. The model will be built taking into account various factors, such as age, income, and occupation, that are known to influence people's buying habits. These factors are known as predictor attributes. The output that is predicted is called the target attribute, and its values (whether the person will buy the camera or not) are known as categories or classes. Once the model has been generated, it can be applied to other data to come up with a prediction. This is known as model apply or scoring. In our example, you would use the model to predict if a certain customer is likely to buy a digital camera.

In this example, the target attribute has two values—will buy a digital camera and will not buy a digital camera. You can also use classification to predict attributes with more than two values—for example, whether the risk of a person defaulting on a payment is low, medium, or high.

Classification is often used to create customer profiles. For instance, once you have determined which of your customers are likely to buy a digital camera, you can then profile them by occupation, as shown in Figure 9.38. From this graph, you can see that most likely buyers are either engineers or executives. So you can now target your promotions more accurately toward these customers and reduce your costs.

Click To expand
Figure 9.38: Classification.

In order for classification to work well, the build data must contain enough samples for each target category; otherwise, the results may not be accurate. In other words, your build data must include enough people who have bought digital cameras in the past and enough who have not.

Testing a classification model

Data mining using classification usually involves a testing phase to check how good the model is. For this, data where the outcome is known is tested to see how well the model's predictions match the outcome. For instance, you could take data for customers who have bought a digital camera in the past and check this data against the predictions given by the model.

Testing a model involves computation of a structure known as the confusion matrix. A confusion matrix tells you how many times the model's prediction matched the actual data and how many times it did not. The columns correspond to the predicted values and the rows to the actual values. For instance, in Figure 9.39, the model was correct 555 + 45 = 600 times and wrong 12 + 8 = 20 times. This shows that this model is a pretty good one.

Click To expand
Figure 9.39: Confusion matrix.
Computing Lift

Another metric used to determine the effectiveness of a model, is its lift. To understand what lift means, consider the following example. Suppose we have a customer base of 100,000 households and on an average about 2.5 percent (2,500 customers) respond to any given promotion. We would like to get smarter and only target those customers who are most likely to respond. With a good classification model, we should be able to identify most of the likely 2,500 respondents by targeting much fewer than the 100,000 households. Given a certain percentage of target customers, lift is the ratio of the number of respondents obtained with the model against the number obtained without the model.

Lift is computed as follows. The classification model is applied to an actual customer data set where we know who responded to a past promotion and who did not. The customers are then sorted by their likelihood to respond as predicted by the model, with the most likely respondents first. This sorted list is then divided into ten equal groups known as deciles. For each decile, the number of customers in the data set that had actually responded to the promotion is counted. If the model is any good, most of the respondents should come from the top few deciles, since they were predicted to be the most likely respondents.

If you draw a graph with deciles one through ten on the X-axis and the number of actual respondents on the Y-axis, you will typically get the curve as shown in Figure 9.40. This curve tells you that you only need to target the customers in the first three deciles (30 percent of households) to get 70 percent of those likely to respond. On the other hand, the straight line in this figure corresponds to a random promotion, where everyone is predicted to be equally likely to respond. Without the model to guide you, if you target 30 percent of households, you will only get 30 percent of the likely respondents. To get 70 percent of the likely respondents, you will have to target 70 percent of households! The higher the curved line is from the straight line for the first one or two deciles, the better the model's lift.

Click To expand
Figure 9.40: Using lift analysis for targeted promotions.

Confusion matrix and lift are both widely employed techniques to determine the accuracy of classification models. Oracle data mining provides APIs that will allow you compute these quantities.

9.2.2 Oracle data mining algorithms

Now that we have seen the techniques that are used in data mining, we will briefly look at the various algorithms offered by Oracle for each of these techniques.

Association rules

The algorithm used by Oracle data mining for association rules is called Apriori. The user provides the minimum support and confidence desired. The algorithm first finds single items that occur frequently and have the minimum support—for example, fish. It then finds pairs of items that have the minimum support such that at least one item in the pair was itself frequently occurring—for example, fish and lemons. It repeats the process to come up with increasingly larger combinations of items until it can find no more. Once it has found all frequent item sets, it finds those among them that satisfy the minimum confidence requirement from the user. These are reported as association rules.

Classification

Oracle offers two main algorithms for classification: Naïve Bayes algorithm and Adaptive Bayes Network algorithm.

The Naïve Bayes (NB) algorithm is based on a probability theorem known as Bayes theorem and assumes that each attribute is independent from the other. An interesting property of the NB algorithm is that you can build and cross-validate the model using the same data. This algorithm works best with a small number of predictor attributes (less than 200).

The Adaptive Bayes Network (ABN) algorithm produces a model in the form of a decision tree, such as that shown in Figure 9.41. From this decision tree you can see that men between the ages of 15 and 35 and women over 26 are likely to buy a camera. Since the model produced by ABN is in a human-readable form, a business analyst or executive would be more comfortable when using it to make a business decision. The ABN algorithm is also more accurate than NB, but it takes longer to build the model.

Click To expand
Figure 9.41: Adaptive Bayes Network decision tree.

In addition, Oracle also offers a Model Seeker feature. This feature allows you to generate multiple models using the ABN or NB algorithms and then determines which is the best with regards to a specific target category. For instance, it could tell you which model best predicts people who may buy a digital camera, or which model best predicts people who will buy kitchen appliances. Model Seeker also generates a summary of all the models it built so that the user can determine which model is the best.

Clustering

Oracle data mining supports two algorithms for clustering: Enhanced k-means and O-Cluster.

The k-means algorithm is a clustering algorithm that groups data into a specified number of k clusters. It groups items into clusters based on their relative distance from each other. So, all points in one cluster are closer to each other than to points in other clusters. The Enhanced k-means algorithm is a variation on the k-means algorithm, which forms clusters in a hierarchical fashion. It starts with all the data being in one cluster and then successively splits this data into smaller clusters until the desired number of clusters is obtained. It is very efficient compared with traditional k-means, since it only requires one pass through the data and hence can handle large data sets. It works well even for data sets with less than ten attributes. The metric distance used by k-means can only be defined for numerical attributes; hence, if you have discrete values (e.g., color = red, black, blue), k-means cannot be used.

The O-Cluster algorithm defines clusters using ranges of attribute values. The user does not need to provide the number of clusters to generate. This algorithm can be used for nonnumeric attributes that have a discrete set of values.

9.2.3 Data preparation in Oracle data mining

In order to use data mining, you may need to preprocess the data to reduce its size or complexity. Binning and attribute importance are two techniques commonly used.

Binning

Data mining algorithms typically require that the data be "binned" into a discrete set of buckets for analysis. Oracle provides the capability to automatically perform binning or allows a user to manually bin the data by specifying the boundary values for the buckets.

Attribute importance

If your data has a large number of predictor attributes, it can be a challenge to determine which ones influence your required target attribute and which ones do not. For instance, age may determine a customer's preference in music. The customer's height or the color of his or her eyes may have little impact. Depending on the number of predictor attributes, the classification process can be quite time consuming to compute. Attribute importance is a technique that automatically ranks attributes by how likely they are to influence the target attribute. This helps you narrow down the relevant predictor attributes before building the classification model.

The algorithm for attribute importance in Oracle data mining is known as predictor variance. It takes as input a data set, along with its predictor and target attributes, and ranks the predictor attributes. It finds how the target attribute changes with changes in each predictor attribute (i.e., computes the variance of the target attribute with respect to the predictor attribute). If the variance is high, it means that the predictor attribute has a significant influence on the target attribute and is ranked higher.

9.2.4 The PMML standard

The Predictive Modeling Markup Language is an emerging XML-based standard to define data mining models. PMML provides a vendor-independent method of defining models so that the models can be exchanged between different applications. Thus one application may produce a model, and another may apply the model to a data set. Oracle data mining supports import and export of association rules and Naïve Bayes models.

9.2.5 Using Oracle data mining

Oracle Data Mining consists of two components: Oracle Mining Server and Oracle data mining Java API.

Oracle Mining Server is a component within the Oracle database that performs the mining tasks. The Mining Server consists of a repository, which stores the tables containing inputs and outputs of the analysis.

Oracle data mining API is a Java API, which allows an application developer to write data mining programs for the techniques discussed earlier.

Installation and configuration

Oracle data mining (ODM) is available as part of the Oracle 9i installation using the Universal Installer. During installation, if you choose to create a preconfigured database (see Chapter 2), ODM is automatically installed for you. You must, however, unlock the ODM and ODM_MTR accounts that came with the preconfigured database and set new passwords, as follows:

alter user odm identified by <odm_password_here> account unlock;
alter user odm_mtr identified bu <odm_password_here> account unlock;

If you have chosen to build a custom database, you must use the Database Configuration Assistant (DBCA) to install the ODM option and follow its instructions.

It is recommended that you use partitioning (if available) and parallel execution for data mining tasks, especially for large data sets, since they offer significant performance gains.

To use data mining functionality, the ODM Task Monitor must be running. To start the ODM Monitor, log in as the ODM user and issue the following command in SQL*Plus:

execute ODM_START_MONITOR;

To stop the monitor, issue the following command:

execute ODM_STOP_MONITOR;

Now you are ready to start using the data mining APIs.

Data mining tasks

A typical flow of a data mining analysis will involve the following steps:

  1. Preprocessing the input data for analysis: The raw data may need to be preprocessed before analysis. Some algorithms require that the data be binned. The user can preprocess the data and specify how this data is to be binned. This is optional, since Oracle can automatically perform binning, if the user has not done so.

  2. Specifying the data to be mined: The data to be mined must be placed in a single table. Each fact in the data being analyzed is known as a case. For example, "customer #3, male, aged 35, a teacher by profession, bought a digital camera" is one case. In the table, a case may be represented using either a single record or multiple records—one for each attribute, as illustrated in Figure 9.42.

    Click To expand
    Figure 9.42: Table formats for data mining.

    If all attributes for a case are placed in the same row, the table is said to be in a nontransactional format. Alternatively, the table could store each case using multiple records with the format (customer#, attribute_name, attribute_value). This is known as a transactional format.

    Regardless of how the data is stored, Oracle will convert the data into a transactional format prior to analysis.

    The input table specification is defined using a Physical Data Specification (PDS) Java object.

  3. Setting up parameters for the analysis: Next, you must indicate what function you would like to use (association, clustering, classification, or attribute importance). You can specify parameters at two levels—at the level of the function or technique you want to use and at the level of the actual algorithm you want to use for that technique. If you do not specify the algorithm setting, Oracle will use a default. The function settings for analysis are stored in a Mining Function Settings (MFS) Java object. The actual API to use to create the MFS depends on the mining technique you are using. We will see an example of classification in the next section. An important part of MFS settings is the Logical Data Specification, where you specify how each mining attribute has to be treated by the mining algorithm. This will include the attribute type, as in whether the attribute is categorical or numerical, the use, as in whether it is a target attribute or a predictor, and its preparation status, as in whether the attribute is prepared or unprepared.

    After the mining function settings object is created, it is a good idea to validate it to ensure it is correct. You can also persist the settings object in the database by giving it a name. This allows you to reuse the same settings in different programs.

  4. Building a model: Now you are ready to build a model. To build a model, you create a MiningBuildTask object, make it persistent in the database, and then execute it. The model is executed asynchronously. You can query the status of the task during execution. When execution is complete, the model will be stored in the database.

  5. Applying the model to new data: Once you have built a model, you can apply it to new data to make predictions. You can apply a model on new cases stored in a table or to a single case. The new data must be preprocessed (if needed) and stored in a table compatible with the one used to build the model. To apply the model, you first describe the location of the new data, as in step 2. Next, you specify the location for the output table where the results or predictions must be placed and identify what subset of attributes should be placed in the output table. Finally, to apply the model, you create a MiningApplyTask object, give it the name of the model you described earlier, and execute it. The results will be stored in the database in the specified output table.

  6. Testing a model and computing lift: As described earlier, when using classification you may want to test the model and compute its lift. To test a model and compute its lift, you must specify the location of the test data, which must have the same format as the build data. To test the model create a ClassificationTestTask object and execute it. A confusion matrix is generated and stored in a ClassificationTestResult object. To compute lift, create a Mining-ComputeLiftTask and execute it. The lift values for each decile are stored in a MiningLiftResult object.

Now we will illustrate some of these steps using the Oracle Data Mining APIs.

9.2.6 An example using Oracle data mining APIs

We will present an example of using the APIs for one technique, namely, classification. This will illustrate the general flow of the data mining analysis. (Note: The detailed description of all the APIs is beyond the scope of this book. You can find the entire API specification with your Oracle installation.)

Example 1: Building a model.

We will build a model for classification to predict if a person is likely to buy a digital camera. The input data set is stored in a table, EASYDW.TEST_BUILD_DATA. The data is stored in a nontransactional format, meaning that all attributes for a case are stored in the same row of the table. The table definition is shown in the following code segment. The target attribute, CAMERABUYER, indicates whether the person has bought a digital camera or not. It has the value 1 if the person has bought a digital camera and 0 if not. The example is in the form of snippets of Java code.

SQL> describe test_build_data;
 Name                                    Null?    Type
---------------------------------------- -------- ---------------
CUSTOMERID                                        NUMBER(10)
AGE                                               NUMBER
SEX                                               VARCHAR2(8)
OCCUPATION                                        VARCHAR2(10)
CAMERABUYER                                       NUMBER(1)

  1. Preprocessing the input data for analysis: We will not pre-process the data and let Oracle automatically perform binning.

  2. Specifying the mining server to use: We first create an instance of a DataMiningServer object, specifying the database to connect to using a JDBC URL for the user to log in as. On successful login, we get a connection handle (dmsConn), which will be used in the subsequent APIs.

    DataMiningServer dms = new
 DataMiningServer
   ("jdbc:oracle:thin:@slawande-pc:1521:orcl",
    "odm", "odm");

oracle.dmt.odm.Connection dmsConn = dms.login();

  3. Specifying the structure of data: We must first create a LocationAccessData object, which indicates the schema and table name of the input data. Then we create a PhysicalDataSpecification object of the NontransactionalDataSpecification type. (If the table were in transactional format, you would use TransactionalDataSpecification instead.)

    LocationAccessData lad = new
   LocationAccessData("TEST_BUILD_DATA", "EASYDW");
PhysicalDataSpecification pds = new
   NonTransactionalDataSpecification(lad);

  4. Setting up parameters for the analysis: Next, we specify parameters to do classification. We will let Oracle determine the algorithm to use and not specify any algorithm-level settings. The DataPreparationStatus uses a value of "unprepared" to indicate that the data has not been prebinned, and Oracle should apply automatic binning.

    The target attribute is specified as CAMERABUYER and is of type categorical, because it has discrete values.

    ClassificationFunctionSettings cfs
 = ClassificationFunctionSettings.create
     (dmsConn, null, pds, "camerabuyer",
      AttributeType.categorical,
      DataPreparationStatus.getInstance
        ("unprepared"));

    If we want to use the Naïve Bayes algorithm, we must also define the algorithm-level settings and pass them into the function-level settings. The code would then look something like the following. Notice that we have passed the Naïve Bayes settings variable nbs into the ClassificationFunctionSettings class.

    NaiveBayesSettings nbs
  = new NaiveBayesSettings(0.01f, 0.01f);

ClassificationFunctionSettings cfs
= ClassificationFunctionSettings.create
    (dmsConn, nbs, pds, "targetattr",
    AttributeType.categorical,
    DataPreparationStatus.getInstance
      ("unprepared"));

    Before we proceed with building the model, we will verify the settings and store them in the database under the name TEST_DATA_CFS.

    cfs.validate();
cfs.store(dmsConn, "TEST_DATA_CFS");

  5. Building a model: To build a model, we create a MiningBuildTask object and store it persistently in the database under the name TEST_DATA_CFS_MODEL. Then we will execute the task and wait for it to complete.

    MiningBuildTask task = new
   MiningBuildTask(pds, "TEST_DATA_CFS",
                   "TEST_DATA_CFS_MODEL");

task.store(dmsConn, "TEST_DATA_CFS_TASK");
task.execute(dmsConn);
MiningBuildStatus = task.waitForCompletion(dmsConn);

    The resulting model is stored in the database. You can see the persistent information about various MiningFunctionSetting, MiningTask, and MiningModel objects by querying various metadata tables in the ODM schema, as follows:

    SELECT name, user_name, function_name, algorithm_name
FROM odm_mining_function_settings;

NAME           USER_NAME  FUNCTION_NAME    ALGORITHM_NAME
-------------  ---------- -------------    --------------
TEST_DATA_CFS  odm        classification   naiveBayes

SELECT id, name, type, job_id FROM odm_mining_task;

ID   NAME                TYPE    JOB_ID
---  ------------------  -----   ------
24   TEST_DATA_CFS_TASK  BUILD   24

SELECT name, function_name, algorithm_name, data_location
FROM odm_mining_model;

NAME                 FUNCTION_NAME    DATA_LOCATION
-------------------  --------------   ---------------------
TEST_DATA_CFS_MODEL  classification   EASYDW.TEST_BUILD_DATA

    The next example shows how this model is applied to new data.

Example 2: Applying a model

In this example, we will apply the model we built in the previous example to predict customers likely to buy a digital camera product.

  1. Specifying the mining server to use: This step is the same as in the previous example.

  2. Specifying the logical structure of data: Next, we specify the location of the data to apply the model to. This table must have a format compatible with the build data. In our example, we will use the table EASYDW.TEST_APPLY_DATA.

    LocationAccessData lad = new
   LocationAccessData("TEST_APPLY_DATA", "EASYDW");
PhysicalDataSpecification pds = new
   NonTransactionalDataSpecification(lad);

  3. Specifying the location of the output: On applying the model, the resulting predictions will be placed in a table. We must specify the location of this table and also a MiningApplyOutput object describing what columns we would like in it. In our example, we will use the table EASYDW.TEST_APPLY_OUT.

    LocationAccessData ladOut = new
   LocationAccessData("TEST_APPLY_OUT", "EASYDW");

MiningApplyOutput mao = new MiningApplyOutput();

    Each row of the output table contains a prediction for the target attribute, in our case whether the customer is likely to buy a camera or not. The target attribute is represented by a prediction (1 or 0), a probability for the prediction, and a rank.

    We define the target attribute using an ApplyTargetProbability-Item object, as follows:

    ApplyTargetProbabilityItem camerabuyerAttr = new
  ApplyTargetProbabilityItem
   (new Attribute("CameraBuyer", DataType.stringType),
    new Attribute("Probability", DataType.stringType),
    new Attribute("Rank", DataType.stringType));

    The camerabuyer attribute has two values: 1, meaning the customer is a camera buyer, and 0, meaning the customer is not. We must define these values, as follows:

    camerabuyerAttr.addTarget
  (new Category("CameraBuyer", "1",
                DataType.getInstance("int")));

camerabuyerAttr.addTarget
  (new Category("NotCameraBuyer", "0",
                DataType.getInstance("int")));

    In addition, we would like to store the customerid for whom this prediction was made in the output table. This is copied from the source table, EASYDW.TEST_APPLY_DATA, to which the model is being mined, and, hence, we define it using an Apply-SourceAttributeItem object, as follows. The first parameter specifies the name and properties of the column in the source table, and the second parameter specifies the name of this column in the output table.

    ApplySourceAttributeItem customerIDAttr = new
  ApplySourceAttributeItem
    (new MiningAttribute("CUSTOMERID",
                         DataType.intType,
                         AttributeType.notApplicable,
                         false, false),
     new Attribute("CUSTOMER_ID", DataType.intType));

    Finally, we add these to the MiningApplyOutput object we created previously.

    mao.addItem(camerabuyerAttr);
mao.addItem(customerIDAttr);

  4. Creating and executing a mining apply task: Now we are ready to apply the model. This is nearly identical to building a model. We create a MiningApplyTask and supply it with the location of the input data, the name of the model (one we created in the previous example), the location of the output table, and the mining apply output specification. We then store this in the database under the name TEST_DATA_CFS_APPLYTASK and execute it.

    MiningApplyTask task = new
   MiningApplyTask(pds,
                   "TEST_DATA_CFS_MODEL",
                   mao, ladOut,
                   "TEST_DATA_CFS_OUTPUT");

task.store(dmsConn, "TEST_DATA_CFS_APPLYTASK");
task.execute(dmsConn);

    Once the task finishes, the predictions are stored in the EASYDW.TEST_APPLY_OUT table. Notice that the columns correspond to the attributes we defined in step 3. Each customerid had two rows: one with the probability of being a camera buyer (prediction = 1), and the other with the probability of not being a camera buyer (prediction = 0). The predictions are ranked according to which is the more likely case. Thus, customerid 4 is unlikely to buy a camera (rank= 1 => camera buyer = 0), while customerid 26 is more likely to buy one (rank = 1 => camera buyer = 1).

    SELECT * FROM EASYDW.TEST_APPLY_OUT;

CAMERABUYER PROBABILITY       RANK CUSTOMERID
----------- ----------- ---------- ----------
       0  .959025771          1          4
       1  .040974229          2          4
       0  .699846267          1         20
       1  .300153733          2         20
       1  .573198618          1         26
       0  .426801382          2         26
...

You can use any of the analysis tools provided by Oracle, such as 9i Discoverer or Reports to analyze this result. Most of the diagrams shown in this chapter have been obtained by analyzing results of various mining functions using Discoverer. In some cases, a small amount of programming may be needed to retrieve the results into tables.


Summary

9.3 Summary

Data Mining is a process that finds hidden trend patterns in your data. Oracle data mining embeds data mining functionality in the database server, thereby improving the mining performance and allowing it to scale to large amounts of data. In this chapter, we looked at various applications of data mining and how they can be done using Oracle data mining. We hope this chapter has given you some appreciation of how data mining can be used to make better business decisions.


Appendix

This book describes how to use a wide range of Oracle products. To use the software as described in this book, you will need the following products. The version mentioned below is the minimum software version of the product that supports the described features, at the time of writing. It is recommended that you use the latest version of the product available to you.

Oracle Database Enterprise Edition, 9.2.0.1.0

  • Oracle Data Mining Option 9.2.0.1.0

  • Oracle OLAP Option 9.2.0.1.0, for most OLAP functionality

  • Oracle OLAP Option 9.2.0.2.0, for Analytic Workspace Creation Tools

Oracle 9i Application Server, Enterprise Edition, Release 2, 9.0.2

  • JDeveloper 9.0.3 with Business Intelligence Beans

  • Oracle 9i Developer Suite (for Discoverer, Report Services, and Warehouse Builder)

The above software can be downloaded at http://otn.oracle.com. To obtain a listing of the SQL examples in the book, please go to the website http://www.lilianhobbs.com. You can also contact the authors by e-mail at <ssl@cs.wisc.edu> or <susan.hillson@oracle.com>.


index_A

Index

A

Access parameters, specifying, 250–51
Adaptive Bayes Network (ABN) algorithm, 482
AGGMAP command, 443, 445
Aggregate rollup, 172–73
with hierarchy described by dimension object, 173
occurrence, 172
See also Query rewrite
Aggregates, computing, 176–77
Aggregation
capabilities, 102
functions for, 102–9
map, 443
Allocation, 430
ALTER INDEX REBUILD statement, 377
ALTER TABLE EXCHANGE PARTITION clause, 270–71
Analytical applications, 429–31
allocation, 430
analytical calculations and aggregation, 429–30
defined, 429
financial calculations, 430
forecasting, 430
modeling, 430
what-if analysis, 430–31
Analytical workspaces, 433, 434–38
accessing, in applications, 446–50
concepts, 435–38
creation tools, 457–60
defined, 428, 434–35
persistent, 435
temporary, 435
Analytic Workspace Creation Wizard, 459–60
Analytic Workspace Manager, 457
defined, 458
illustrated, 458
Applications
accessing analytical workspaces in, 446–50
analytical, 429–31
launching, 343–44
Apriori, 481
Association rules, 475–76
algorithm, 481
confidence, 476
defined, 475
illustrated, 476
support, 475
using, 476
See also Data mining; Oracle data mining
ATTRIBUTE clause, 165
defined, 179
in dimensions, 179–80
Automatic DML monitoring, 380
Automatic memory management, 101–2
Availability, 16–17


index_B

Index

B

Backup configuration, 345–47
creating, 345–47
defined, 345
selecting channels in, 346
selection, 348
Backup(s), 27, 344–56
configuration, creating, 345–47
content lists, 360
custom strategy, 351–52
directory, 346
file sizes, 355–56
frequency, 348
full, 347
incremental, 352–53
job in library, 352
level, 352–53
predefined strategy, 347–50, 351–52
procedure definition, 349
procedure review, 350
RAM, 356
report files needing, 361
storing on tape RMAN, 347
tablespace, 354–55
types of, 345
BI Beans, 434, 460–73
data access, 461
defined, 460
tasks, 460
Binning, 483
Bitmap indexes, 70–73
alternative, 76
bitmap join index vs., 75
compression, 72, 92
creating, in OEM, 73
defined, 70
disadvantages, 72–73
on foreign-key columns, 99
occupation, 71
recommendations, 299
sex, 70
on two columns, 71
using, 72
See also Indexes
Bitmap join indexes, 73–76
bitmap index vs., 75
bitmaps, 74
defined, 73
example, 73–74, 75
See also Indexes
B*tree indexes, 69–70
defined, 69
index-key, 69–70
key compression, 92
rowids, 69
space requirements, 70
varieties, 69
See also Indexes
Bucketing functions, 113–14
Buffer Cache Advisor, 383
Business area, 307–9
defined, 307
in Discoverer Administrator, 318
granting access to, 317
naming, 309
setting up, 309
Business Intelligence Beans. See BI Beans


index_C

Index

C

Calculation Builder, 470-73
Calculations, 470-73
analytical, 429-30
creating, 471
financial, 430
graphing, 472
parameters specification, 472
report totals, 417
Capacity Planner, 383
CASE expression, 127-28
defined, 127
searched, 127
use example, 127-28
uses, 128
Change data, 212-20
capture (CDC), 213-20
changes, transporting, 220
identifying, 212-13
processing, 217-20
publishing, 214
publish/subscribe architecture, 214
querying, 218
subscribing, 216
subscription creation, 216-17
subscription termination, 220
See also Extracting data
Change set, 214
Change tables, 213
Classification, 478-81
algorithms, 482
building model for, 487-90
confusion matrix, 479, 480, 481
for customer profile creation, 478-79
defined, 478
illustrated, 479
model testing, 479
process, 478
See also Data mining; Oracle data mining
Cleansing data, 221-22
combining/separating and, 222
transformations for, 261-62
Clustering, 477-78
algorithms, 482-83
defined, 477
illustrated, 477
in market segmentation, 478
performing, 478
See also Oracle data mining
Columns
cardinality, 70
foreign-key, 99
visible restriction, 309-10
width (reports), 418
Composite partitioning, 82-85
defined, 82
illustrated, 82
process, 83-84
with SUBPARTITION TEMPLATE clause, 84
types of, 82
uses, 85
See also Partitioning
Compression, 92-94
attribute, changing, 94
bitmap index, 92
b*tree index, 92
key, 92
segment, 92-93
CONCAT operator, 244, 440
Conditional report, 420-21
Confusion matrix, 479, 480, 481
Constraints, 43-51
arguments against, 44
check, 48, 63, 65
creating, 61-65
defined, 43
disabling, 44, 242
Easy Shopping Inc., 61-65
enabling, without validation, 246
foreign, 47
null, 46
primary-key, 180-81
recording, 49
reenabling, 241, 243
referential integrity, 238
relationship defined by, 181
rewrite and, 180-82
system tables, querying, 49
validation, 49
viewing, 50
violations, handling, 245
Control file, 228-29
data definition language, 229
defined, 228
illustrated, 229
role, 228
See also SQL*Loader
Cost-based optimizer, 94, 95
defined, 95
query rewrite and, 186-87
CREATE DIMENSION statement, 162, 163, 167
CREATE MATERIALIZED VIEW statement, 140, 147-48
CREATE OUTLINE statement, 385
CREATE TABLE AS SELECT statement, 272-73
Cube
creation, 456
defined, 434, 450
properties, editing, 456
See also OLAP metadata
CUBE operator, 102, 103
defined, 102, 103
output comparison, 104
with supported aggregates, 103
time/space consumption, 104
See also SQL functions
Cube Viewer, 456
Cube Wizard, 453-56
dimensions selection, 453
fact tables selection, 453
joining dimensions and fact tables, 454
measures specification, 455
summary screen, 455
CUME_DIST() function, 113, 124
Custom backups, 351-52
CWM packages, 450-51


index_D

Index

D

Data
access, external tables, 251–52
aggregating, 442–43
change, 212–20
cleansing, 221–22, 261–62
compression, 92–94
deriving, 222, 261–62
drilling on, 304, 325
extracting, 211–20
filtered, 174–76, 294
generators, 295
integrating, 221
loading, 209–78
moving, 258, 268–73
multidimensional, SQL access to, 448–49
operational, 7
partitioning, 53–54, 76–91
publishing, 277
purchasing, 8
structure specification, 488
transforming, 7, 220–25
unstructured, 16
validating, 262
volume, increasing, 16
Database Configuration Assistant (DBCA), 484
Databases
backing up, 277
creating, with Oracle Enterprise Manager, 34–36
features, 34
file location, 33
naming, 31
Oracle 9i, 432–33
preconfigured, 31
restoring, 361–64
selecting, 349
single vs. many decision, 29–30
standby, 366
storage, 35
virtual private, 390–92
Data-driven formatting, 466–68
illustrated, 467
rules specification, 467
Data files
adding, 38–41
Easy Shopping Inc., 60–61
location, 38
location specification, 248–49
Data file (SQL*Loader), 230
Data marts, 8–11
data warehouse vs., 25
dependent, 10, 11
independent, 9
Data mining, 493
algorithms, 474
defined, 13, 473
Oracle, 13–14, 473–92
statistical and machine learning techniques, 473
table formats, 485, 486
uses, 474
Data warehouse design, 19–66
aggregation decision, 20
constraints, 43–51
database number decision, 29–30
data files and tablespaces, 38–41
dimensional modeling, 21–22
dimension table, 23
dimension table creation, 41–43
Easy Shopping example, 25–26
entity-relationship modeling and, 20–21
fact table, 22
fact table creation, 41–43
fundamental flaw, 368
generating/configuring, 297–300
implementing, 29–58, 297
inclusion decision, 20
indexes, 51–52
introduction, 19–20
for management, 26–28
materialized views, 55
naming conventions, 30
normalization, 24–25
Oracle Database Configuration Assistant and, 30–36
partitioning, 53–54
for performance, 28–29
schemas, 36–38
security, 56–57
testing, 59
validating, 295–96
warehouse keys, 23–24
Data warehouses
application types, 17
automatic memory management, 101–2
availability, 16–17
challenges, 14–16
database design and, 4–5
data marts vs., 25
data volume, 16
defined, 1
e-business intelligence and, 426
enterprise, 11
evolution as separate systems, 5–8
features, 67–133
for finding information, 4
future, 17–18
historical perspective, 2–11
incrementally building, 10
indexing, 68–76
loading data into, 209–78
managing, 14–15, 329–98
multiple subjects, 8
need for, 1–2
normalizing, 24–25
from operational systems, 6–8
optimizations, 67
Oracle software for, 401–2
partitioning, 76–91
real-time, 17–18
refreshing, 378
reorganizing, 368–78
rise of, 3–5
schemas, 26
tuning, 382–89
users/performance, 17
Web and, 399–425
See also Warehousing tools
DBMS_AW package, 447–48
DBMS_MVIEW refresh procedures, 149–51
DBMS_OLAP package, 200–206
DBMS_STATS, 66, 154
statistics collection with, 379
use of, 154
Decision-support applications, 67
Decision support systems (DSS), 4
DENSE_RANK() function, 110
Deriving data, 222, 261–62
Dimensional modeling, 21–22
defined, 21
illustrates, 22
Dimensions, 139, 162–69, 435–36
ATTRIBUTE clause in, 179–80
choosing, 453
choosing (Presentation Wizard), 462
creating, 162–63
creating (OWB), 287–88
defining, 441
defining, with attributes, 165–66
defining, with multiple hierarchies, 164–65
defining, with normalized tables, 166–67
geography, 162–63
joining with fact tables, 454
level-based, 435–36
loading, 253–55
member selection, 463
normalized, 166
OLAP metadata, 450
relationship definitions, 162
relationships declared by, 178
rewrite with, 177–80
time, 164, 178
time, creating (OWB), 288
validating, 167–69
validating data with, 262
value, 435
variables and, 440–41
verifying, 273
See also Summary Management
Dimension tables
creating, 41–43
defined, 23
Easy Shopping Inc., 61–65
fact table vs., 23
loading, 225–26
Dimension Wizard, 287
Directed learning, 475
Direct path insert
defined, 272
moving data between tables with, 272
Direct path load
defined, 230
parallel, 241–43
of single partition, 237–41
using, 235–36
See also SQL*Loader
DISABLE QUERY WRITE clause, 143
Disaster recovery, 365–68
defined, 365
Enterprise, 367–68
Oracle Data Guard, 366–67
plan, 366
See also Recovery
Discoverer Administrator
automatic joins, 308
business area access, 317
business area illustration, 318
business area naming, 309
business area setup, 309
calculated item creation, 312
changing item details, 311
defined, 300
first-time start, 305
hierarchy definition, 313
item class, 314
join definition, 311
schema selection, 307
summary creation, 315
summary refreshment, 316
Summary Wizard, 315, 316
tables/views selection, 308
task list, 306
See also Oracle Discoverer
Discoverer Desktop, 300
Discoverer Plus, 318–28
data display selection, 320
defined, 300
defining totals, 323
drilling up/down, 325
Export Wizard, 411
format headings, 322
graph of monthly sales, 327
Graph Wizard, 326
launching, 318–19
list of databases for connection, 319
month-level report, 326
report customization, 324
report illustration, 323
report using conditions, 328
sorting data, 322
specific data selection, 328
SQL Inspector, 324–25
table layout, 321
using, 409–10
with workbook, 320
See also Oracle Discoverer
Discoverer Viewer
advantages, 301
category sales, 302
defined, 300
presentation options (screen 1), 303
presentation options (screen 2), 304
presentation options (screen 3), 305
query execution, 302
querying with, 301–3
report criteria specification, 410
report from browser, 409
starting, 301
using, 409–10
See also Oracle Discoverer
Dynamic sampling, 381


index_E

Index

E

Easy Shopping Inc., 25-26, 36
constraint creation, 61-65
data files, 60-61
index creation, 61-65
schema, 59-67
security definition, 65-66
table creation, 61-65
tablespaces, 39, 60-61
E-business intelligence, 426
ENABLE/DISABLE clause, 44, 45
ENABLE QUERY REWRITE clause, 170
ENABLE QUERY WRITE clause, 143, 144
End-user layer (EUL), 305-6
Entity-relationship modeling, 20-21
Environment setup (Oracle Discoverer), 305-18
business area, 307-9
end-user layer, 305-6
folders, 310
hierarchies, 312-13
item classes, 313-14
item details, changing, 310
items, creating, 311-12
security, 317-18
summaries, 314-16
visible column restriction, 309-10
ETL process, 6
development difficulty, 210
extraction, 211-20
illustrated, 210
load, 225-59
tools, 277
transformation, 220-25
See also Loading data
Events, 336-38
advising users of, 338
creating, 336
naming, 336
notification schedule for, 338
selecting, 337
using, 336-38
See also Management
Exchange partitions, 370
Executive information systems (EIS), 12
EXPLAIN_MVIEW utility, 157-61
defined, 157-58
on existing materialized view, 161
functions, 158
invoking, 158
in Oracle Enterprise Manager, 161
output, 158-59
query rewrite and, 160
EXPLAIN PLAN, 95-96, 191, 192, 253
EXPLAIN_REWRITE utility, 193-96
defined, 193
results, 194
using, 194
EXTEND_WINDOW procedure, 218
External tables, 247-55
access parameters specification, 250-51
creating, 249-51
data access, 251-52
data file/log files location, 248-49
defined, 247
illustrated, 249
loading data from, 252
loading data in parallel with, 252-53
metadata specification, 249
read access, 248
See also Tables
Extracting data, 211-20
change data identification, 212-13
defined, 211
Oracle change data capture, 213-20
See also ETL process
Extraction/transformation/load. See ETL process


index_F

Index

F

Fact tables, 22
choosing, 453
creating, 41–43
creating (OWB), 288–89
defined, 22
dimension table vs., 23
Easy Shopping Inc., 61–65
information, 22
joining with dimensions, 454
loading, 226
moving data from staging table to, 268–73
partitioned, 268
surrogate keys in, 224
Fast refresh, 152–57
defined, 152
example, 152–53
good performance, 154
materialized views and, 152
with partition maintenance operations, 154–57
situations, 160
See also Refreshing; Refresh operation
FCEXEC command, 444
FCSET command, 444
Filtered data, 174–76
Filters, 197–98, 201
Financial calculations, 430
FIRST function, 121
Folders
defined, 310
measure, 450
Forecasting, 430
defined, 430
OLAP DML for, 443–45
Full backups, 347
Functional dependencies, 162, 165


index_G

Index

G

Global indexes, 88-89
defined, 88
illustrated, 89
SQL code for creating, 89
See also Partitioned indexes
GRANT command, 389
Graphical user interface (GUI) tools, 29
Graphs, 468-70
data display, 468
drill-down capability, 470, 471
example, 470
illustrated, 469
layout specification, 469
Graph Wizard, 326
GROUP BY clause, 111, 172
in materialized view query, 144
query rewrite with, 190-91
GROUPING function, 107-9
behavior, 107
defined, 107
uses, 108
values, 107
See also SQL functions
GROUPING_ID function, 108-9
GROUPING SETS function, 104-7
capability, 105
concatenated, 106
defined, 102
multiple, specifying, 106
with supported aggregates, 103
See also SQL functions
Groups
creating, 334-35
defined, 334
illustrated, 335


index_H

Index

H

Hash join, 97–98
Hash partitioning, 79–80
data skew and, 79
defined, 79
example, 80
illustrated, 79
uses, 80
See also Partitioning
HAVING clause, 111
Hierarchical relationships, 162
Hierarchies, 312–13


index_I

Index

I

Incremental backups, 352–53
defined, 352
level, 352–53
selecting, 353
See also Backup(s)
Independent data marts, 9
Indexes, 28, 51–52
bitmap, 70–73
bitmap join, 73–76
b*tree, 69–70
building block, 68
changes, 377–78
drop, 242
Easy Shopping Inc., 61–65
global, 88–89
local, 90–92
maintenance, deferring, 236
Oracle Enterprise Manager, 52
partitioned, 88–92
partitions, 372
planning, 52
selection, 51
types of, 51, 68
unstable, checking for, 246–47
unstable, rebuilding, 247
Indexing, 68–76
decisions, 68
materialized views, 147
needs, understanding, 68
Index Tuning Wizard, 383
INSERT SELECT statement, 248
INSERT statement, 298
Integrating data, 221
Internet, 399–401
Intranet, 399–401
contents example, 400
defined, 399
Inverse percentile, 123–24
Items
calculated, creating, 312
classes, 313–14
creating, 311–12
defined, 310
details, changing, 310–11


index_J

Index

J

JavaServer pages, 402
Java servlets, 402
Jobs
backup, 352
creating, 339
logging from, 342
number of, 343
output from, 342
parameters for, 340
scheduling, 338-43
selecting tasks for, 340
storage, 339
watching, run, 341
Job Scheduler, 338-43
Join-back, 173-74
defined, 173
using, 173-74
See also Query rewrite
Joins
defining (Discoverer), 311
hash, 97-98
lossless, 181
Mapping Editor with, 292
nested loops, 97
optimization, 97-101
partition-wise, 98
sort-merge, 97
star transformation, 99-101


index_K

Index

K

Key lookup feature, 293–94
Kimball, Ralph, 5
K-means algorithm, 483


index_L

Index

L

Lag and lead functions, 109, 120-21
defined, 120
example, 120
uses, 121
See also SQL functions
LAST function, 121
Level-based dimensions, 435-36
Lift
computing, 480-81
defined, 480
See also Oracle data mining
LIMIT command, 441
Linear regression analysis, 122
LIST BACKUP command, 359
LIST command, 359
List partitioning, 80-81
defined, 80
example, 81
illustrated, 81
See also Partitioning
Loading data, 209-78
dimensions, 253-55
dimension tables, 225-26
direct path load of single partition, 238-39
with duplicate dates, 237
ETL process and, 209-11
with external tables, 247-55
fact table, 226
initial population, 209
in parallel, 237
parallel direct load, 242
postload operations, 273-77
in single partition, 237
with SQL*Loader, 226-46
with transportable tablespaces, 256-60
See also Data warehouses; ETL process
Load Wizard, 227-35
accessing, 228
illustrated, 227
load monitoring, 233
load scheduling, 232
SQL*Loader advanced options, 232
SQL*Loader modes of operation, 231
See also Oracle Enterprise Manager; SQL*Loader
Local indexes, 90-92
defined, 90
nonprefixed, 91-92
partitions of, 374
prefixed, 90-91
types of, 90
See also Partitioned indexes
Lossless join, 181


index_M

Index

M

Management, 329–98
design for, 26–28
events and, 336–38
groups and, 334–35
issues, 395–98
job scheduling and, 338–43
launching applications and, 343–44
methods, 329
with OEM, 329–44
outline, 385
parallel, tasks, 381–82
partitions facilitate, 376–77
security, 389–92
space-use monitoring, 392–95
summary, 135–207, 398
tasks, 26
tasks, evaluating/practice, 397
Mapping Editor, 290–92
illustrated, 291
with joins, 292
opening, 290–91
Mappings
defining, 290–95
source to target, 290–91
See also Oracle Warehouse Builder
Market-basket analysis, 475
Massively parallel processor (MPP) systems, 133
Materialized views
contents specification, 143–44
creating, 139–48
creating, with PREBUILT clause, 145
creating in Oracle Enterprise Manager, 141
for data replication, 139
defined, 138, 139
definition parts, 140
enabling, for query write, 143
fast refresh and, 152
indexing, 147
monitoring use, 205–6
multiple, query rewrite with, 191–93
ON COMMIT refresh, 142–43
ON DEMAND refresh, 143, 149
optimizer statistics, 276–77
partitioning, 146–47
physical storage, 141
population with data, 141–42
recommending, 202
refreshing, 274–76
refreshment method, 142
security of, 147–48
size, estimating, 206
stale, looking for, 274
Matrix report, 420
MAX function, 118
Measure folders, 450
Measures
choosing (Presentation Wizard), 462
defined, 434, 450
specifying, 455
See also OLAP metadata
MERGE statement, 254, 255, 298
Merging partitions, 371–73
illustrated, 372
index partitions and, 372
See also Partitions
Metadata
defined, 15
exporting, 258–59
importing, 259–60
OLAP, 450–555
OWB, 280
role of, 15
specifying, 249
Modeling, 430
dimensional, 21–22
E-R, 20–21
Models
applying, 490–93
building, 487–90
OLAP API multidimensional, 449
Modules
creating, 282–84
defined, 281
Moving data, 268–73
with exchange partition, 270–71
between tables with direct path insert, 272
Moving partitions, 376
Moving window aggregate functions, 109, 115–18
Multicolumn partition keys, 86–88
Multidimensional (MOLAP)
defined, 427
products, 428
ROLAP vs., 427–28
See also On-line analytical processing (OLAP)
Multitable insert, 266


index_N

Index

N

Naive Bayes (NB) algorithm, 482
Naming
business area, 309
conventions, 30
databases, 31
events, 336
Nested loops join, 97
New Module Wizard, 282-83
Nonprefixed local indexes, 91-92
Normalization, 24-25
NOVALIDATE clause, 49, 181
NTITLE() function, 114-15


index_O

Index

O

Object Management Group (OMG), 15
Object privileges, 56
O-Cluster algorithm, 483
OLAP API, 428, 434, 449-50
defined, 434, 449
multidimensional model, 449
OLAP catalog, 434, 450-57
defined, 450
requirements, 450
OLAP DML, 433, 438-46
for aggregation, 442-43
commands, issuing, 440
custom menu creation, 445-46
defined, 438
executing, 439
for forecasting, 443-45
formulae definition, 445
power of, 442
reporting data with, 441-42
SQL and, 439
OLAP metadata, 450-57
creating, in Oracle Enterprise Manager, 451-57
cube, 450, 453-56
definition methods, 450
dimensions, 450, 452, 454
entities, 450
measure folders, 450
measures, 450, 455
OLAP Summary Advisor, 456
OLAP_TABLE function, 448-49
On-line analytical processing (OLAP), 4
calculation engine, 433
defined, 427
multidimensional (MOLAP), 427-28
queries, 13, 427
relational (ROLAP), 427-28
tools, 14
See also Oracle 9i OLAP
On-line transaction processing (OLTP), 2
E-R techniques, 5
optimal performance, 5
in Oracle applications, 18
process automation, 3
ON PREBUILT TABLE clause, 144
Operating System Backup Utility, 344
Operational data store (ODS), 9-10
Operational system keys, 223
Optimizations, 67
cost-based, 94, 95
join, 97-101
query, 94-101
rule-based, 94-95
Optimizer statistics
automated gathering, 380
collected with DBMS_STATS package, 379
gathering, 378-81
for materialized views, 276-77
for tables, 273
Oracle 9i, 11-14
database, 432-33
defined, 11
indexes, 28
parallel DML, 131
Partition Change Tracking (PCT), 155-57
preconfigured databases, 31
products, 11
RAC technology, 133-34
Release 2, 428
Summary Advisor, 196-206
Summary Management, 55, 135-207
Oracle 9i Application Server, 402-12
defined, 402
development, 402
HTTP Server, 402
Oracle Clickstream, 407-8
Personalization, 14, 407
products, 403
Oracle 9iAS Portal, 405-7
benefits, 405-6
customized pages, 406
defined, 405
integrating reports into, 412
Oracle 9i Reports integration, 406-7
Oracle 9i Backup and Recovery Manual, 347
Oracle 9i Database, 12-14
data mining functionality, 13-14
express server technology convergence, 13
features, 12
Oracle 9i OLAP, 428-29
analytical workspaces, 433, 434-38
architecture, 431-34
BI Beans, 434, 460-73
components, 431
OLAP API, 434, 449-50
OLAP calculation engine, 433
OLAP catalog, 434, 450-57
OLAP DML, 433, 438-46
Oracle 9i database, 432-33
Release 2, 432
table functions, 433-34
Oracle 9i Option, 428-29
defined, 428
Discoverer and, 429
Oracle 9i Reports, 412-25
components, 413
conditional report, 421
defined, 412
examples, 419-20
matrix report, 420
Oracle 9iAS Portal integration, 406-7
Queue Manager, 423
Report Wizard, 413-19
in Web format, 423
Oracle 9i Reports Server, 423-25
job queue, 425
output format specification, 424-25
report schedule, 425
report selection, 424
Oracle Clickstream, 407
defined, 408
information, 408
Oracle Database Configuration Assistant, 30-36
database features, 34
database storage, 35
defined, 30
file location variables, 33
initialization parameters, 32
Oracle Data Guard, 366-67
configuration, 366, 367
defined, 366
Oracle data mining, 13-14, 473-93
algorithms, 481-83
analysis steps, 485-87
APIs, 487-93
association rules, 475-76
classification, 478-81
clustering, 477-78
data preparation in, 483-84
defined, 474
installation/configuration, 484-85
lift analysis, 480-81
predicator variance, 484
problem categories, 474-75
summary, 493
table formats, 485, 486
tasks, 485-87
techniques, 474-81
using, 484-87
See also Data mining
Oracle Data Mining Suite, 474
Oracle Discoverer, 300-328
Administrator, 300
business area, 307-9
defined, 300
Desktop, 300
dynamic reports, 303
elements, 300
environment setup, 305-18
EUL, 305-6
folders, 310
hierarchies, 312-13
item classes, 313-14
item details, changing, 310-11
items, 310
items, creating, 311-12
Oracle 9i OLAP Option and, 429
Oracle Portal integration, 301
Plus, 300
report types, 301-5
security issues, 317-18
summaries, 314-16
use of, 301
Viewer, 300, 301-3
visible column restriction, 309-10
Oracle Enterprise Manager
administrator account, 333
check constraints, 48
constraints, 45
create a user, 37
creating databases with, 34-36
Cube Wizard, 453-56
dimensions, 40
dimensions for OLAP in, 452
EXPLAIN_MVIEW in, 161
foreign key, 47
index creation in, 73
indexes, 52
Job Scheduler, 338-43
Load Wizard, 227-35
in looking for stale materialized views, 274
managing with, 329-44
materialized view creation in, 141
null constraint, 46
object security, 57
OLAP metadata creation, 451-57
OLAP worksheet in, 439
schemas, 38
for summary refreshment, 150, 151
tablespaces, 39
Table Wizard, 42-43
three-tier architecture, 330
viewing constraints, 50
viewing table partitions, 270
Oracle Enterprise Manager Console, 330-33
elements, 331
illustrated, 331
launching, 330
launching applications, 343-44
Navigator tree, 332
Navigator window, 332-33
scheduling jobs, 338-43
starting, for first time, 331
Oracle Expert, 383
Oracle managed files (OMF), 393
Oracle Warehouse Builder (OWB), 8, 279-300
benefits, 280
bitmap index recommendations, 299
Client, 281-82, 283
code generation, 298
data generators, 295
data source definition, 284
data sources, 283
defined, 279
design generation/configuration, 297-300
design validation, 295-96
dimension creation, 287-88
Dimension Wizard, 287
fact table creation, 288-89
filtering data, 294
functions, 280
key lookup, 293-94
manual table definition, 286
Mapping Editor, 290-92
metadata, 280
module creation, 282-84
New Module Wizard, 282-83
physical design configuration, 299
repository, 280, 281
Repository Assistant, 280
setup, 280-81
setup steps, 281
source to target mapping definition, 290-95
table definition, 285-87
table import, 285
time dimension creation, 288
transformations, 289-90
validation progress, 296
Warehouse Fact Table Editor, 288-89
ORDER BY clause, 110-11, 124
Outlines, 384-89
creating, 384-85
defined, 384
editing, 386-89
management, 385
original, 386
revised, 388
using, 386
See also Plan stability


index_P

Index

P

PARALLEL clause, 58
Parallel direct path load, 241–43
defined, 241
disable constraints/triggers, 242
drop indexes, 242
illustrated, 242
inspect log, 243
load data, 242
reenable constraints/triggers, 243
See also Direct path load
Parallelism, 130–34
defined, 130
degree, setting, 132
hardware requirements, 133–34
operations benefiting from, 131
Parallel management tasks, 381–82
Parallel queries, 131
PARTITION BY clause, 111–13, 118
defined, 111
illustrated, 112
rank calculations with, 113
Partition Change Tracking (PCT), 155–57, 377
defined, 155
illustrated, 155
query rewrite and, 187–89
refresh, 155, 157
stale materialized views and, 157
support, 157
Partitioned indexes, 88–92
global, 88–89
local, 90–92
See also Indexes
Partitioning, 53–54, 76–91
benefits, 76
composite, 82–85
criteria, 54
defined, 53
hash, 79–80
illustrated, 53
list, 80–81
materialized views, 146–47
performance and, 77
process, 76–88
range, 77–79
techniques, 53
by time, 77
Partition keys
defined, 76
multicolumn, 86–88
for range partitioning, 86–87
Partitions
adding/dropping, 369–70
exchanging, 370
facilitate management, 376–77
hash, coalescing, 375
index, 372
list, 375
of local indexes, 374
maintenance, 369–77
merging, 371–73
moving, 376
range, 375
splitting, 371, 373–75
truncating, 375–76
Partition-wise join optimization, 98
PERCENTILE_CONT() function, 123–24
PERCENTILE_DISC() function, 123–24
PERCENT_RANK() function, 113, 124
Performance dependencies, 67
design for, 28–29
fast refresh, 154
partitioning and, 77
query, 382
SQL*Loader, 235–37
Performance Manager, 383
Plan stability, 384–89
defined, 384
outline creation, 384–85
outline edit, 386–89
outline use, 386
Portals
data warehouses and, 404
defined, 403
Oracle example, 404
reasons for using, 403–7
Portlets, 406
Postload operations, 273–77
database, table, partition backup, 277
dimension verification, 273
materialized views, refreshing, 274–76
optimizer statistics for materialized views, 276–77
optimizer statistics for tables, 273
publish data, 277
Preconfigured databases, 31
Predicator variance, 484
Predictive Modeling Markup Language (PMML) standard, 484
Prefixed local indexes, 90–91
PREPARE_SUBSCRIBER_VIEW procedure, 219
Presentation Wizard, 461–66
crosstab layout, 463
dimension member selection, 463
for graph creation, 468–70
illustrated, 461
measures and dimensions selection, 462
report formats, 461
Privileges
ALTER ANY MATERIALIZED VIEW, 148
CREATE MATERIALIZED VIEW, 147–48
defining, 65–66
Easy Shopping Inc., 65–66
object, 56
system, 67
Projects, 281


index_Q

Index

Q

Queries
computing aggregates in, 176-77
conditions, 176
defining, 461
with multiple groupings, 191
with multiple materialized views, 191-93
OLAP, 13, 427
performance, 382
response times, determining, 398
specifying, 415
Query Builder, 416, 457, 461
Query optimization, 94-101
cost-based, 94
join, 97-101
rule-based, 94-95
Query rewrite, 170-96
aggregate rollup, 172-73
constraints and, 180-82
defined, 138
with dimensions, 177-80
enabling, 170
filtered data, 174-76
with GROUP BY clause, 190-91
grouping sets and, 190-93
integrity dependence, 186
integrity modes, 182-86
join-back, 173-74
with multiple groupings, 191
with multiple materialized views, 191-93
optimizer and, 186-87
PCT and, 187-89
problems, troubleshooting, 193-96
query identification, 171
SQL text match, 171-72
transformation support, 170
See also Summary Management
QUERY-REWRITE_INTEGRITY mode, 182-86
ENFORCED, 182, 183, 184, 185
STALE_TOLERATED, 182, 183, 185
TRUSTED, 182, 185, 186
Query Wizard, 465


index_R

Index

R

RAID technology, 92
Range partitioning, 77–79
defined, 77
in EASYDW schema, 77–79
illustrated, 78
partition keys, 86–88
See also Partitioning
RANK() function, 110, 112, 124
Ranking functions, 110–15
CUME_DIST(), 113
defined, 109, 110
examples, 118, 119, 120
NTITLE(), 114–15
PERCENT_RANK(), 113
RANK(), 110, 112
WIDTH_BUCKET(), 115
See also SQL functions
RATIO_TO_REPORT function, 119–20
Real application clusters (RAC), 16–17
buffer cache sharing, 134
defined, 134
technology, 133–34
Real-time data warehouses, 17–18
Recommendations, 198–200
graph, 198
implementing, 199–200
materialized views, 202
view/modify, 199
See also Summary Advisor
Recovery
configuration, 363
disaster, 365–68
log, 364
Recovery catalog, 356–57
creating, 357
database placement, 357
database registration with, 359
illustrated, 357
Recovery Manager (RMAN), 329, 344
back contents, 360
backups, 356
Recovery Wizard, 361–63
report files needing backup, 361
running, 358
schema report, 360
Recovery Wizard, 361–63
configuration, 363
location of data files, 363
starting, 362
Refreshing, 137–38
complete, 138
defined, 137
incremental, 138
materialized view, 142–43
options, 275
with Oracle Enterprise Manager, 150–51
warehouses, 378
See also Summary tables
Refresh operation, 149–57
DBMS_MVIEW procedures, 149–51
defined, 149
fast, 152–57
of multiple materialized views, 150
ON COMMIT, 142–43
ON DEMAND, 143, 149
PCT, 155, 157
policy, 149
Relational OLAP (ROLAP)
defined, 427
MOLAP vs., 427–28
See also On-line analytical processing (OLAP)
Relations, 436
RELY clause, 45, 182
Reorganizing, 368–78
index changes, 377–78
partition maintenance, 369–77
reasons for, 368–69
REPORT command, 359, 441
Reporting aggregates, 109, 118–20
MAX, 118
RATIO_TO_REPORT, 119–20
SUM, 118, 119
See also SQL functions
Reports
changing, 325
column widths, 418
conditional, 420–21
with conditions, 328
creating, with Presentation Wizard, 461–66
creating, with Report Wizard, 413–19
customizing, 324
data source, 414
defining, 412
Discoverer Plus, 323
Discoverer Viewer, 409, 410
distributing, to Web, 410–12
dynamic, with graphs, 303
fields, selecting, 417
formats, 423
grouping levels, 416
illustrated, 419
integrated from Discoverer into Oracle 9iAS Portal, 412
layout, 418
matrix, 420
medium, selecting, 413
month-level, 326
output format, 424
run-time format, 423
scheduling, 425
selecting, 424
style, selecting, 414
Summary Advisor, 202–4
total calculations, 417
for Web, 421–25
in workbook, 318
REPORT SCHEMA command, 359
Report Wizard, 413–19
column widths specification, 418
data source selection, 414
grouping levels specification, 416
Query Builder setting, 416
report field selection, 417
reporting medium selection, 413
report style selection, 414
SQL query specification, 415
totals calculation, 417
See also Oracle 9i Reports
Repository, 280
location, 281
tablespace, 281
Repository Assistant, 280
Restoring, 361–64
concerns, 361
with Recovery Wizard, 362–64
type, selecting, 362
Resumable statements, 394–95
status, 395
suspended, 394, 395
REVOKE command, 389
Roles, 56–57
allocating, 390, 391
creating, 390
EASY_USER, 389
Rolling window operations
adding/dropping partitions to perform, 369–70
defined, 369
ROLLUP function, 102, 103–4
defined, 102, 103
groupings determination, 14
output comparison, 104
with supported aggregates, 103
See also SQL functions
Rule-based optimizer, 94–95


index_S

Index

S

Scheduling jobs, 338–43
Schemas, 36–38
defined, 36
Easy Shopping Inc., 59–67
multiple, 36
name, 37
objects, 36
star, 25, 26
Scripts, Summary Advisor, 202–4
Security, 56–57
Easy Shopping Inc., 65–66
level, 389
maintaining, 389–92
of materialized views, 147–48
object privileges, 56–57
roles, 56–57
system privileges, 57
Sort-merge join, 97
Space
automated management, 393
resumable allocation, 394–95
use monitoring, 392–95
utilization, 393
Splitting partitions, 371, 373–75
defined, 371
illustrated, 374
See also Partitions
SQL Analyze, 383
SQL functions, 102–30
for aggregation, 102–9
for analytical computing, 109–25
CUBE, 102, 103
CUME_DIST(), 113, 124
DENSE_RANK(), 110
FIRST, 121
GROUP BY, 102, 104–7
GROUPING, 107–9
GROUPING_ID, 108–9
inverse percentile, 123–24
lag and lead, 109, 120–21
LAST, 121
linear regression, 122
MAX, 118
moving window aggregate, 109, 115–18
NTITLE(), 114–15
PERCENTILE_CONT(), 123–24
PERCENTILE_DISC(), 123–24
PERCENT_RANK(), 113, 124
RANK(), 110, 112, 124
RATIO_TO_REPORT, 119–20
reporting aggregates, 109, 118–20
ROLLUP, 102, 103–4
statistical, 109, 122
SUM, 118, 119
user-defined aggregates, 125–27
WIDTH_BUCKET(), 115
SQL Inspector, 324–25
SQL*Loader, 226–46
advanced options, 232
constraint violations, 245
control file, 228–29
conventional path load, 230
data definition language, 229
data file, 230
direct path load, 230
direct path load (parallel), 241–43
direct path load (single partition), 237–41
enabling constraints without validation, 246
error handling, 231–32
load exception processing, 244
log inspection, 233–35, 240–41, 243, 244, 397
modes of operation, 230–31
monitoring load operation progress, 233
performance optimization, 235–37
postload operations, 244–47
scheduling the load, 232
transformations with, 243–44
unstable indexes, checking for, 246–47
unstable indexes, rebuilding, 247
See also Loading data
SQL MERGE
defined, 253
loading dimensions with, 253–55
SQL text match, 171–72
Staging table, 268–73
Stale materialized views, 157, 274
Stale summaries, 137
Star schemas
defined, 26
illustrated, 25
Star transformation, 99–101
availability, 99
defined, 99
functioning of, 100–101
Statistical functions, 122
defined, 109
linear analysis, 122
SUBPARTITION TEMPLATE clause, 84, 85
Subscriber views
defined, 214
preparing, 219
Subscriptions
creating, 216–17
ending, 220
handle, 216
SUM function, 118, 119
Summary Advisor, 196–206, 383
with DBMS_OLAP package, 200–206
defined, 138
filters, 197–98, 201
hypothetical workload, 205
materialized view size, estimating, 206
materialized view use, monitoring, 205–6
recommendations, 198–200, 202
reports and scripts, 202–4
results, purging, 205
run methods, 197
run requirements, 196
with Wizard, 197–200
workload, 197–98, 200–201
See also Summary Management
Summary Advisor Wizard, 197–200
defined, 197
filters, 198
implement recommendations, 199
loading the workload, 197
recommendation graph, 198
view/modify recommendations, 199
Summary Management, 55, 135–207
components, 138–39
defined, 135
dimensions, 139, 162–69
with existing summary tables, 144–46
materialized views, 138, 139–48
need for, 135–38
query rewrite, 138, 170–96
Summary Advisor, 138, 196–206
Summary tables, 55, 135–39
complete refreshed, 138
defined, 135
existing, Summary Management with, 144–46
illustrated, 136
incrementally refreshed, 138
questions, 138
refreshed, 137–38, 149–57
stale, 137
Summary Wizard, 315, 316
Super Administrator, 333
Supervised learning, 475
Surrogate keys, 23, 222–24
in fact table, 224
uses, 223
Symmetric multiprocessor (SMP) systems, 133
System privileges, 57
System statistics, collecting, 381


index_T

Index

T

Table functions, 264–66, 433–34
defined, 264
invoking, 266
object technology, 264
OLAP_TABLE, 448–49
TRANSFORM, 265
Tables
backing up, 277
change, 213
creating, 61–65
data mining formats, 485, 486
dimension, 23, 41–43
external, loading with, 247–55
fact, 22, 23, 41–43, 268–73
hash-partitioned, 375
joining, to obtain data, 292–93
nontransactional format, 485
normalized, 166
optimizer statistics for, 273
parent-child dimension, 436
partitions, viewing, 270
staging, 268–73
summary, 135–39
transactional format, 485
Tablespace backup, 354–55
Tablespace Map, 383
Tablespaces, 38–41
altering, to read only, 258
altering, to read/write, 260
creating, 237, 256, 269
creating from SQL, 40
creating with Oracle Enterprise Manager, 39, 41
defined, 39
Easy Shopping Inc., 39, 60–61
names, 41
repository, 281
transportable, 256–60
Table Wizard, 42–43
final screen, 50, 51
partitioning criteria, 54
partitioning screen, 53–54
primary key creation within, 45–46
Testing
design, 59
new software, 396–97
system, building, 396
Timing data loads, 397
TopSessions, 383
Transformation(s), 220–25
choosing place to perform, 224–25
for cleansing data, 261–62
data cleansing, 221–22
data derivation, 222
data integration, 221
defined, 220
for deriving data, 261–62
during load process, 225
inside Oracle Server, 260–73
OWB, 289–90
as part of extraction process, 224
splitting data source into multiple targets, 266–68
with SQL*Loader, 243–44
in staging area, 224–25
warehouse key generation, 222–24
See also ETL process
Transportable tablespaces, 256–60
alter tablespace to read only step, 258
alter tablespace to read/write step, 260
create tablespace step, 258
defined, 256
export metadata step, 258–59
illustrated, 257
import metadata step, 259–60
move data step, 258
steps, 256
transport tablespace step, 259
uses, 256
See also Tablespaces
Triggers, 213
disabling, 238, 242
reenabling, 241, 243
Truncating partitions, 375–76
Tuning, 382–89
as black art, 383–84
defined, 382
plan stability, 384–89
tools, 383–84


index_U

Index

U

Undirected learning, 475
Unstable indexes, 246–47
checking for, 246–47
rebuilding, 247
Unsupervised learning, 475
Usage patterns, 15
User-defined aggregates, 125–27
defined, 125–26
in extensibility framework, 126
implementation, 127
ODCIAggregate interface, 126
SalesForecast() function, 126–27
See also SQL functions
USER_EXTERNAL_TABLES dictionary view, 251


index_V

Index

V

VALIDATE_DIMENSION procedure, 167, 169
Validating
components, 295
data, 262
design, 295-96
progress, 296
Value dimensions, 435
Variables
conceptual view, 437
defined, 437
dimensions and, 440-41
Virtual private database (VPD), 390-92
defined, 392
for fine-grained access control, 392


index_W

Index

W

Warehouse Fact Table Editor, 288–89
Warehouse keys, 23–24
generating, 222–24
looking up, 262–63
operational system, 223
surrogate, 222–24
Warehouse Module Editor, 286, 287
Warehousing tools, 279–328
choosing, 279
Oracle Discoverer, 300–328
Oracle Warehouse Builder (OWB), 279–300
Web
data published on, 400
data warehousing and, 399–425
distributing reports to, 410–12
functions, 400–401
report building for, 421–25
See also Internet; Intranet
What-if analysis, 124–25, 430–31
WHEN clause, 267
WHERE clause, 111, 143, 174, 392
WIDTH_BUCKET() function, 115
Window aggregate functions, 109, 115–18
WITH clause, 129–30
benefits, 129
defined, 129
use example, 129–30
Workbook, 318, 320
Workload
defined, 197
filter, 197–98
hypothetical, 205
loading, 197, 200
user-defined, 200–201


List of Figures

Chapter 1: Data Warehousing

Figure 1.1: Enterprise data warehouse with dependent data marts.

Chapter 2: Designing a Warehouse

Figure 2.1: Dimensional modeling.
Figure 2.2: Normalized versus dimensional normal form.
Figure 2.3: Dimensional model for Easy Shopping Inc.
Figure 2.4: Oracle Database Configuration Assistant—naming the database.
Figure 2.5: Oracle Database Configuration Assistant—initialization parameters.
Figure 2.6: Oracle Database Configuration Assistant—file location variables.
Figure 2.7: Oracle Database Configuration Assistant—database features.
Figure 2.8: Oracle Database Configuration Assistant—database storage.
Figure 2.9: Oracle Enterprise Manager—create a user.
Figure 2.10: Oracle Enterprise Manager Console—schemas.
Figure 2.11: Oracle Enterprise Manager Console—tablespaces.
Figure 2.12: Oracle Enterprise Manager Console—dimension.
Figure 2.13: Oracle Enterprise Manager Console—Table Wizard.
Figure 2.14: Oracle Enterprise Manager Console—table columns.
Figure 2.15: Oracle Enterprise Manager Console—constraints.
Figure 2.16: Table Wizard—specifying the primary key.
Figure 2.17: Oracle Enterprise Manager—null constraint.
Figure 2.18: Oracle Enterprise Manager—foreign key.
Figure 2.19: Oracle Enterprise Manager—check constraints.
Figure 2.20: Oracle Enterprise Manager—viewing constraints.
Figure 2.21: Table Wizard— final screen.
Figure 2.22: Oracle Enterprise Manager—indexes.
Figure 2.23: Partitioning.
Figure 2.24: Table Wizard—partitioning screen.
Figure 2.25: Table Wizard—partitioning criteria.
Figure 2.26: Oracle Enterprise Manager—object security.

Chapter 3: Data Warehouse Features

Figure 3.1: Creating a bitmap index in Oracle Enterprise Manager.
Figure 3.2: Range partitioning.
Figure 3.3: Hash partitioning.
Figure 3.4: List partitioning.
Figure 3.5: Range-hash composite partitioning.
Figure 3.6: Global index.
Figure 3.7: Prefixed local index.
Figure 3.8: Nonprefixed local index.
Figure 3.9: PARTITION BY clause.
Figure 3.10: Parallel query.

Chapter 4: Summary Management

Figure 4.1: Creating a materialized view in Oracle Enterprise Manager.
Figure 4.2: Refreshing a materialized view in Oracle Enterprise Manager.
Figure 4.3: Refreshing a materialized view in Oracle Enterprise Manager—Options.
Figure 4.4: Partition change tracking.
Figure 4.5: EXPLAIN_MVIEW in Oracle Enterprise Manager
Figure 4.6: Bubble diagram for the geography dimension.
Figure 4.7: Bubble diagram for the time dimension.
Figure 4.8: A normalized dimension.
Figure 4.9: Summary Advisor Wizard—loading the workload.
Figure 4.10: Summary Advisor Wizard—filters.
Figure 4.11: Summary Advisor Wizard—recommendation graph.
Figure 4.12: Summary Advisor Wizard—view/modify recommendations.
Figure 4.13: Summary Advisor Wizard—implement recommendations.
Figure 4.14: Summary Advisor report.

Chapter 5: Loading Data into the Warehouse

Figure 5.1: The ETL process.
Figure 5.2: Change data capture publish/ subscribe architecture.
Figure 5.3: Querying change data.
Figure 5.4: The use of surrogate keys in the warehouse.
Figure 5.5: The Load Wizard.
Figure 5.6: Accessing the Load Wizard.
Figure 5.7: The control file.
Figure 5.8: SQL*Loader modes of operation.
Figure 5.9: SQL*Loader advanced options.
Figure 5.10: Scheduling the load.
Figure 5.11: Monitoring jobs from the OEM console.
Figure 5.12: SQL*Loader parallel direct path load.
Figure 5.13: External table.
Figure 5.14: Transportable tablespaces.
Figure 5.15: Multitable insert.
Figure 5.16: Data is moved from a staging table into the fact table.
Figure 5.17: Viewing a table's partitions using enterprise manager.
Figure 5.18: Exchange partition.
Figure 5.19: Using Oracle Enterprise Manager to look for stale materialized views.
Figure 5.20: Refresh options.
Figure 5.21: Explain materialized view.
Figure 5.22: Analyzing the materialized view after refresh.

Chapter 6: Oracle Warehousing Tools

Figure 6.1: Warehouse Builder—steps to setup.
Figure 6.2: Warehouse Builder—creating a module.
Figure 6.3: Warehouse Builder—type of data source.
Figure 6.4: Warehouse Builder—data source defined.
Figure 6.5: Warehouse Builder—two modules created.
Figure 6.6: Warehouse Builder—import a table.
Figure 6.7: Warehouse Builder—manually define a table.
Figure 6.8: Warehouse Builder—Warehouse Module Editor.
Figure 6.9: Warehouse Builder—dimension hierarchy.
Figure 6.10: Warehouse Builder—fact table creation.
Figure 6.11: Warehouse Builder—Fact Table Editor.
Figure 6.12: Supplied functions for transformations.
Figure 6.13: Mapping Editor.
Figure 6.14: Mapping Editor with joins.
Figure 6.15: Product key lookup.
Figure 6.16: Filtering source data.
Figure 6.17: Validating the design.
Figure 6.18: Validation progress.
Figure 6.19: Implementing the design.
Figure 6.20: Code generated by Oracle Warehouse Builder.
Figure 6.21: Configuring the physical design.
Figure 6.22: Bitmapped index recommendations.
Figure 6.23: Discoverer Viewer—executing our query.
Figure 6.24: Discoverer Viewer—category sales by month report.
Figure 6.25: Discoverer Viewer—presentations options (screen 1).
Figure 6.26: Discoverer Viewer—presentations options (screen 2).
Figure 6.27: Discoverer Viewer—presentations options (screen 3).
Figure 6.28: Discoverer Administrator—create an EUL.
Figure 6.29: Discoverer Administrator—task list.
Figure 6.30: Discoverer Administrator—select the schema.
Figure 6.31: Discoverer Administrator—select the tables and views.
Figure 6.32: Discoverer Administrator—automatic joins.
Figure 6.33: Administrator—naming the business area.
Figure 6.34: Discoverer Administrator—setting up the business area.
Figure 6.35: Discoverer Administrator—change the item details.
Figure 6.36: Discoverer Administrator—define a join.
Figure 6.37: Discoverer Administrator—creating a calculated item.
Figure 6.38: Discoverer Administrator—defining a hierarchy.
Figure 6.39: Discoverer Administrator—item class.
Figure 6.40: Discoverer Administrator—creating a summary.
Figure 6.41: Discoverer Administrator—refreshing the summary.
Figure 6.42: Discoverer Administrator—Summary Wizard.
Figure 6.43: Administrator—granting access to the business area.
Figure 6.44: Discoverer Administrator—business area.
Figure 6.45: Discoverer Plus—list of databases for connection.
Figure 6.46: Discoverer Plus—using a workbook.
Figure 6.47: Discoverer Plus—selecting data to be displayed.
Figure 6.48: Discoverer Plus—table layout.
Figure 6.49: Discoverer Plus—format headings.
Figure 6.50: Discoverer Plus—sorting the data.
Figure 6.51: Discoverer Plus—defining totals.
Figure 6.52: Discoverer Plus—report.
Figure 6.53: Discoverer Plus—customize the report.
Figure 6.54: Discoverer Plus—SQL Inspector.
Figure 6.55: Discoverer Plus—drilling up/down the data.
Figure 6.56: Discoverer Plus—report at the month level.
Figure 6.57: Discoverer Plus—Graph Wizard.
Figure 6.58: Discoverer Plus—graph of monthly sales.
Figure 6.59: Discoverer Plus—select specific data.
Figure 6.60: Discoverer Plus—report using conditions.

Chapter 7: Managing the Warehouse

Figure 7.1: Launching the Oracle Enterprise Manager Console.
Figure 7.2: Oracle Enterprise Manager Console.
Figure 7.3: The Enterprise Manager Navigator window.
Figure 7.4: Creating an Enterprise Manager administrator account.
Figure 7.5: Creating a group.
Figure 7.6: Group—Easy Shopping Inc.
Figure 7.7: Creating an event.
Figure 7.8: Selecting events to monitor.
Figure 7.9: Advising users of an event.
Figure 7.10: Notification schedule for events.
Figure 7.11: Creating a job.
Figure 7.12: Selecting task for a job.
Figure 7.13: Parameters for a job.
Figure 7.14: Scheduling a job.
Figure 7.15: Watching a job run.
Figure 7.16: Logging from the job.
Figure 7.17: Output from the job.
Figure 7.18: Starting applications from the console.
Figure 7.19: Creating a backup configuration.
Figure 7.20: Selecting the channels in a backup configuration.
Figure 7.21: Backup frequency.
Figure 7.22: Backup configuration selection.
Figure 7.23: Selecting the database.
Figure 7.24: Reviewing the backup procedure.
Figure 7.25: Backup Job on the console job queue.
Figure 7.26: Type of custom backup.
Figure 7.27: Including backup job in library.
Figure 7.28: Incremental backup level.
Figure 7.29: Selecting an incremental backup.
Figure 7.30: Selecting a tablespace to back up.
Figure 7.31: Making a tablespace read-only.
Figure 7.32: Backup file sizes.
Figure 7.33: Recovery catalog.
Figure 7.34: RMAN schema report showing the target database.
Figure 7.35: RMAN lists the contents of a backup.
Figure 7.36: RMAN report files needing backup.
Figure 7.37: Selecting type of restore.
Figure 7.38: Location of data files.
Figure 7.39: Recovery configuration.
Figure 7.40: Recovery log.
Figure 7.41: Data guard configuration.
Figure 7.42: Partition maintenance.
Figure 7.43: Merge partition.
Figure 7.44: Split partition.
Figure 7.45: Outline management.
Figure 7.46: The original outline.
Figure 7.47: Changing the join order.
Figure 7.48: Executing the query with a revised outline.
Figure 7.49: Creating a role.
Figure 7.50: Allocating a role to a user.
Figure 7.51: Space utilization.

Chapter 8: Data Warehousing and the Web

Figure 8.1: Signing into the Oracle portal.
Figure 8.2: Oracle portal—standard view.
Figure 8.3: Discoverer Viewer Report from our browser.
Figure 8.4: Discoverer Viewer—specify reporting criteria.
Figure 8.5: Discoverer Plus—Export Wizard—specify output format.
Figure 8.6: Oracle Reports—select the reporting medium.
Figure 8.7: Choosing the style of report and title.
Figure 8.8: Oracle files—selecting the data source.
Figure 8.9: Oracle Reports—specifying the SQL query.
Figure 8.10: Oracle Reports—using the Query Builder.
Figure 8.11: Oracle Reports—specifying the grouping levels.
Figure 8.12: Oracle Reports—selecting the fields in the report.
Figure 8.13: Oracle Reports—calculate totals.
Figure 8.14: Oracle Reports—specify the column widths.
Figure 8.15: Oracle Reports—final report.
Figure 8.16: Oracle Reports—matrix report.
Figure 8.17: Oracle Reports—conditional report.
Figure 8.18: Oracle Reports—Navigator and Designer.
Figure 8.19: Oracle Reports in Web format.
Figure 8.20: Oracle Reports Server—select the report.
Figure 8.21: Oracle Reports Server—specify the output format.
Figure 8.22: Oracle Reports Server—schedule a report.
Figure 8.23: Oracle Reports Server—job queue.

Chapter 9: Oracle 9i OLAP and Data Mining

Figure 9.1: OLAP 9i Release 2 architecture.
Figure 9.2: Level-based versus parent-child dimension tables.
Figure 9.3: Conceptual view of a variable.
Figure 9.4: OLAP worksheet in Oracle Enterprise Manager.
Figure 9.5: Application access to analytical workspaces.
Figure 9.6: Dimensions for OLAP in Oracle Enterprise Manager.
Figure 9.7: Cube Wizard—choosing fact tables.
Figure 9.8: Cube Wizard—choosing dimensions.
Figure 9.9: Cube Wizard—joining dimension and fact tables.
Figure 9.10: Cube Wizard—specifying measures.
Figure 9.11: Cube Wizard—summary screen.
Figure 9.12: Cube Viewer.
Figure 9.13: Analytic Workspace Manager.
Figure 9.14: Analytic Workspace Creation Wizard— Select Cubes.
Figure 9.15: Analytic Workspace Creation Wizard— Select Analytic Workspace.
Figure 9.16: Analytic Workspace Creation Wizard— Enable Workspace for OLAP API & BI Beans.
Figure 9.17: New Presentation Wizard.
Figure 9.18: New Presentation Wizard—choosing measures and dimensions.
Figure 9.19: Layout of a crosstab.
Figure 9.20: Choosing dimension members.
Figure 9.21: The EASYDW sales analysis report.
Figure 9.22: Customizing the content with Query Wizard—Conditions.
Figure 9.23: EASYDW analysis report with top ten products.
Figure 9.24: Customizing the crosstab look and feel.
Figure 9.25: Data-driven formatting.
Figure 9.26: Specifying data-driven formatting rules.
Figure 9.27: Specifying a condition on purchase price.
Figure 9.28: EASYDW analysis report with badly performing products highlighted.
Figure 9.29: Graphs.
Figure 9.30: Specifying the layout of a graph.
Figure 9.31: Graph of product sales by county.
Figure 9.32: Drilling down in the graph.
Figure 9.33: Creating a calculation.
Figure 9.34: Specifying the parameters of the calculation.
Figure 9.35: Graphing a Calculation
Figure 9.36: Association rules.
Figure 9.37: Clustering.
Figure 9.38: Classification.
Figure 9.39: Confusion matrix.
Figure 9.40: Using lift analysis for targeted promotions.
Figure 9.41: Adaptive Bayes Network decision tree.
Figure 9.42: Table formats for data mining.


List of Tables

Chapter 3: Data Warehouse Features

Table 3.1: Sex Bitmap Index
Table 3.2: Occupation Bitmap Index
Table 3.3: Using a Bitmap Index-How many customers are self-employed women?
Table 3.4: Join On Purchases and Customer Tables
Table 3.5: Bitmap Join Index on Purchases and Customer on customer.sex Column

Chapter 4: Summary Management

Table 4.1: Summary Table
Table 4.2: Stale Summary
Team LiB
Previous Section