Many of you have requested I republish my earlier posts and I am happy to do so shortly.

Happy reading…

Remember the AAMIE of security – Authentication, Authorization, Monitoring, Integration, Encryption.

Download the presentation OBIEE-Secured!

And the cost cutting season has begun. We know the drill: across the board budget cuts, new projects delayed, the organization chart de-layered, voluntary retirement packages and, of course, reduction in force and severance packages.

This is not just an exercise in reigning in escalating costs. The more agile organizations also take such opportunities to align their corporate workforce to the new realities of their business and acquire new employees with the skills that are more suited to their  core competencies and evolving business focus.

This brings into focus the need to retrieve and analyze information about employee performance, learning, compensation, benefits, payroll, time management, and recruiting. As with most modules, there are plenty of pre-built HR Analytic applications like Oracle Daily Hr Intelligence, Oracle BI Applications (Siebel Analytics) – HR Analytic module, Cognos Performance Applications, Business Objects HR Analytic Application, to name the leading few. However, more often than not, you will find that these applications work very well if you haven’t customized your HR application (be it Oracle HRMS or SAP R/3 HR module). They will lack some of the most critical information of your business that you have carefully stored in the customizations of the ERP application as I am experiencing while analyzing an Oracle HRMS implementation at one of my clients. After careful evaluation and cost-benefit analyses, we decided to build a custom data mart, rather than buying a pre-built BI application from Oracle or BO. I will focus this series on building the HR data mart from Oracle eBusiness Suite, based on this recent experience.

Building a HR data mart is considerably more difficult than building other common data marts like a Sales data mart or Finance data mart.  The challenges are many, and not all of them are technical. For example, although HR is the second largest (after Finance) provider of information for internal managers, it has a much tighter budget and a smaller cast of analysts to provide you with good analytic requirements. HR analysts are also very wary of sharing “sensitive” information like salary, benefits, and bonuses about the company’s employees. Thirdly, the skills inventory in HR is rarely taken very seriously and hence HR often struggles to gain the focus and attention of senior management for their BI initiatives. From a data warehousing perspective, extracting HR data is more challenging if auditing and date tracking features are enabled in Oracle (which is true over 80% of the time). Moreover, security in HR systems is far more stringent, given the sensitive data that these systems hold. Security in HR goes beyond standard user or responsibility-level security for system access or even restrictions on data visibility based on the organization chart. I will write more about it in a later post, when we talk about the overall design of the data mart.

The HR data mart, particularly in the context of Oracle eBusiness suite has a rather large functional footprint. You need to understand several functional areas / modules to be able to design a data mart that is effective in meeting the reporting / analytic goals of your users. Of particular interest are: HRMS (Core – Employees, Recruiting, etc), Advanced Benefits, Incentive Compensation, iRecuitment, Payroll, Time and Labor, Performance Management, Learning Management and Workforce Scheduling. Although it is not common for a single company to require analytics for all these subject areas, a subset of this (HRMS, Performance Management, and Payroll) is very typical. Most companies have specialized HR reporting needs, for example, on immigration – that are not covered readily in a standard implementation of Oracle HRMS. More on that as we get to the analytics.

Before anything else, here is a quick map of how this series on Human Resources Analytics will be laid out:

Part 1: Overview of Oracle HRMS
Part 2: Key analytics for HR
Part 3: HR data mart design and considerations
Part 4: Source system, ETL tips

Stay tuned…

Case:

ABC Corporation is a manufacturer of small-medium sized office printers. Every printer sold is covered by a 90-day replacement warranty including parts and labor. Beyond that, ABC sells support or maintenance services.

Examples are:

• Standard support service entitles a customer to 12 hrs every week day of phone support and all replacement of defective parts for 1 year. Labor is charged extra. Cost: 5% of equipment value.
• Gold Support service entitles a customer to 24×7 phone support and all replacement of defective parts for 1 year. Labor is included. Cost: 10% of equipment value.

For each order placed by a customer, they record the bill to, ship to, and sold to sites for every order line. These sites could be different. For example, the customer Coca Cola only places orders from its corporate office in Atlanta (SOLD TO), but wants the bills paid by their central finance division in Raleigh (BILL TO), while the items are shipped to their offices in Boston (SHIP TO). ABC attempts to up-sell its “Support service” for every order and, is successful in doing so most of the time. Printer orders usually have a line for support service for each product line. A typical order for 3 printers would look like this:

ABC also installs the equipment after they are delivered. At the time of installation, the exact site of installation (e.g. Boston office, Bldg 1, 3rd floor Loc 1A32) (INSTALLED AT) is recorded. This record also has the serial number of the actual printer. This is now part of the “installed base”.

Installed Base of ABC is a collection of the record of every individual product serial number for each customer’s installed at site. The installed base records for these printers would look like this:

Every installed base record is serviced by one active maintenance contract line at any given point of time. If customers choose to continue the support, these contracts are renewed every year after the warranty period till the life of the product. In the new/renewal contract, the contract number remains the same, but the suffix (_1 in the example below) is incremented. The support prices are incremented by 4% upon each renewal. The service contract related to this order looks like this:

What do we want to analyze? Here is a sample of analytics needed:

• What is the total value of ABC’s installed base? – by customer, by product (not serial #)
• What are the installed base counts (average/min/max/actual printers per customer, etc)
• What is the annuity (annual maintenance contract value)? by customer, by product (not serial #)
• How many customers renew every year (retention rate)? By customers region / D&B attributes, product
• Dollars renewed every year vs. dollars available to renew? By customers region / D&B attributes, product
• Which products tend to stay longer on maintenance? (contracts get renewed)

Each of these analyses is supported by these contracts and there is a requirement to be able to drill down to the contract sub-line level.

If you have worked with the Service Contracts module of Oracle eBusiness suite, all of these will sound very familiar to you. There are some obvious questions that should come up, especially about the grain of the fact tables and the detail of the dimensions – both for products and customers and complexities of the installed base. So, if you are engaged in a similar projects, just trying out this challenge, or if you have questions, do drop me a line or put in a comment.

If you want more details on all the fact and dimension tables for these modules, feel free to drop me a line.

Presentation: How to build BI on Oracle eBS

Refer to part 3 of this series for the broad design. Here are the sources in Oracle eBusiness suite for the different dimension and fact tables:

Dimensions:

Facts and Document Dimensions:

Source Data Structures:

The following diagram outlines the source table relationships for the facts above:

The Budget tables’ relationships are given below:

More detailed diagrams can be downloaded from Oracle Metalink.

The descriptions for the chart of account segments are stored in Flexfields within Oracle. The following query, for example, can be used to get the descriptions:

select gcc.segment1 company
, gcc.segment2 account
, gcc.segment3 cost_center
...
, fvc.description company_desc
, fva.description account_desc
, fvcc.description cost_center_desc
...
from gl_code_combinations gcc
,fnd_flex_values_vl fvc
,fnd_flex_values_vl fva
,fnd_flex_values_vl fvcc
where fvc.flex_value_set_id = (select flex_value_set_id from fnd_flex_value_sets where flex_value_set_name like '<YOUR_COMPANY_GL_AFF_COMPANY')
and fvc.flex_value_meaning = gcc.segment1
and fva.flex_value_set_id = (select flex_value_set_id from fnd_flex_value_sets where flex_value_set_name like ' YOUR_COMPANY_GL_AFF_ACCOUNT')
and fva.flex_value_meaning = gcc.segment2
and fvcc.flex_value_set_id = (select flex_value_set_id from fnd_flex_value_sets where flex_value_set_name like ' YOUR_COMPANY _GL_AFF_COST_CENTER')
and fvcc.flex_value_meaning = gcc.segment3
order by 1,2,3,...
 

ETL Tip:

Loading Fact GL Balances may be tricky if you are not aware that the source table stores balances for all types of amounts – budgets, actual amounts, encumbrances and statistical amounts. Also, it stores the translated amounts if the translation activity has been performed at the close of the fiscal month. As a result of this activity, you may have adjustments posted to certain accounts where you will have actual adjustment amounts posted in the translated currency even though the corresponding local currency amounts are zero.  If you are translating amounts in the Fact GL Balances table on the fly using the reporting tool, be aware of this issue. Although translations can easily be done dynamically, I have found it unwise to do so because of these issues and the complex business rules that often are applied to translations. It is safer to store actual translated values in the fact table itself.

 This is the end of this series. I will write about building Sales Data Marts in the next series. Hope you enjoyed these – I am happy to answer any questions you may have or elaborate on anything I have stated in these posts.

Also read Part 1, Part 2 and Part 3 of this series.

 
Let’s admit it. Most of us who have been in this field for a while have had the sad experience of seeing the latter outcome some time during our careers. Failed projects are harsh realities that most organizations face. We, professionals, learn from these failed initiatives, and apply the knowledge to detect problems early and remedy them effectively.

To understand if your DW/BI initiative is failing, we need to understand what constitutes success. As I noted in my earlier post, An Agile Road to a Successful Data Warehouse, David Wells and James Thomann of TDWI describe three “types” of success and their definitions as they relate to data warehousing – economic success, political success and technical success.
 
Most businesses have moved away from the idea of a large unified data warehouse effort that will involve hundreds of internal resources and consultants and take two years to deliver the first cut of data into the warehouse. In today’s world, you know you have a problem with your initiative if, after a year, you are not seeing some of the benefits you expected. Use this free spreadsheet tool from Nucleus Research to measure ROI for your BI investment.

Signs of a faltering DW/BI initiative may surface in the early stages and these should be taken seriously. The first is missed deadlines and cost overruns for individual projects. It is not unusual for these projects to run two or even three times over budget. Overruns usually result from inflexible business processes that create needless complexity in reports or ETL forcing compromises in design. Costs for such efforts can easily trump whatever benefits they might produce. In a recent project, my client wanted to see costs, revenue and expenses each aggregated to a different grain in one single report, rather than following the much more common practice of drilling down to details from consistent higher level aggregates, all at the same uniform grain. The resultant effort was inordinately expensive to produce a single, unique report. The second warning sign – complaints by employees about poor usability or unmet performance promises may point to serious underlying design problems.

Unfortunately, in many cases, companies tend to just live with such inadequate data warehouses for years on end or, leave them largely unused. A few scrap them altogether. It is not uncommon to find a data warehouse sourcing data from a previously built data warehouse which now merely serves as a repository of data, rather than a central place for analytics. With persistence and patience, such projects can be turned around. The key to uncovering the root cause of the problem is to reassess and reaffirm the business goals of the system and the organizational support it receives. Technical shortcomings can usually be overcome much easier than some of the political hurdles within the company.

Turn-around Team

There are several steps to effectively troubleshoot and recover a stuttering DW/BI initiative. The first step is to assemble a team that can lead the turnaround. This team, led by the chief business sponsor, should comprise of key business partners who have in-depth knowledge of the several business functions the data warehouse aims to benefit. And of course, the team should have competent members of the technical staff, who have the expertise to make different design choices to align the data warehouse to the business goals.

The Rescue Approach

Funding: It is important to have a clear, separate allocation of funds for the attempt to revamp the BI initiative. If you have been able to detect signs of misalignment of the DW/BI initiative in its early stages, then, the chances are you will be able to secure adequate funding for this effort.

Business alignment: Once this team is assembled and empowered to make the bold decisions needed to resurrect the DW/BI initiative, the first task is to reestablish clear business objectives for the initiative. The scope should be tightened to make sure that the most critical gaps are addressed and prioritized and needless features eliminated. Having too broad a scope often leads to a development effort that takes a life of its own and loses touch with the very problem it was meant to solve.

Political alignment: The success of the BI initiative depends on its users. It is important to connect to the very people who will be using the BI system to solve many of their business problems. Without understanding their issues of usability and without granting some political concessions or incentives to secure their buy-in for the initiative, any attempt to revamp the effort will fail. If the culture and practices of the company pose a significant barrier to achieving these objectives, scale back aspirations and build the organizational capacity needed to achieve early results, rather than adopting a strong-arm technique to force adoption of the BI application. The key to success is to identify the most critical users of the BI infrastructure and secure cooperation and collaboration of these users.

Technical realignment: The technical infrastructure, the architecture of the data warehouse, BI metadata and even ETL programs may need radical changes to align these to established best practices. Priorities should be established to deliver analytics that have the greatest business impact, and, to reiterate the point again, the technology architecture should align very closely with the business objectives established by the turn around team.

Monitoring: Once the solution blueprint and implementation roadmap has been adopted, the turnaround team must monitor progress against its reaffirmed or revised goals. As I have written earlier, an agile approach, with frequent releases where the users remain engaged and interested has the greatest chance of success. Nevertheless, it is important to measure this success against internal benchmarks. Make sure project deadlines are met and costs remain under budget and the team should remain alert to any recurrence of the original signs of trouble, such as complaints from users. I believe that a well-executed turnaround ought to yield early wins and measurable progress on most goals within 4 to 6 months, given proper funding and resources.

Ralph Kimball and Margy Ross, in this article – Data Warehouse Check-Ups prescribe periodic assessments of your data warehouse to avoid these common disorders of inconsistent business sponsorship, poor data quality, scant acceptance of the DW/BI environment by the business community, conflicting political priorities of “doing it right” vs. “doing it fast”, and problems with the design or ETL infrastructure. Good reading.

Christening

Hopefully, before embarking on a data warehouse initiative, you have done ample due diligence to form the basis and business justification for such a project. You have probably identified the key benefits you want to derive and the most important business questions you want to answer. You should have also set a not-to-exceed limit of the overall project cost. If you adopt the approach of most product companies, the very next thing you need to do is to establish the identity of the project. Most product companies thrive on acronyms – CDW for Customer Data Warehouse, SEW (Strategic Enterprise Warehouse) and the like. One of my clients, while building an enterprise scale data warehouse, created a theme or analogy of constructing an actual house (the “Great Room” at the centre of the house being the executive dashboard where all key information entities “meet”), and adopted a logo of a construction helmet as well that appeared in all presentations, deliverables, coffee mugs, etc. Such “christening” of the project helps to assimilate the project team to the same mission. It also serves as an introduction to the larger business or user community in the organization and delivers the not-too-subtle message to the business cohorts that they are, in part responsible for the care of the initiative in its infancy. The collaboration with all these business users is where the roots of success of this initiative lie.

Staffing the project

A data warehouse / BI project requires a number of different roles and skills from IS as well as the user communities. It is helpful to remember, that there is hardly a project where you will be able to find a “specialist” for every role of the project. Most projects I have worked have people with multi-faceted skills who can fill more than one role. A word of caution: try to avoid people who claim to have been skilled at every aspect of the project. Such people are less likely to be well versed in best practices in all the different areas and are more likely to cause irreparable damage if entrusted with a key area in which they are deficient, but have masked it well. At one of my previous projects, we had a “Superman Architect” who was not a believer of surrogate keys in the data warehouse. When the warehouse needed to accommodate an additional data source, it faced a tremendous development effort to move from having natural keys to surrogates.  In the real world, you will find people with excellent skills in just two or three areas who can fill those specialist roles.

Let us now examine the different roles needed for a data warehouse / BI project:

Project Manager: The Project Manager needs to be an experienced person, who believes that the project will only be judged by the addition to business value and can maintain the business focus at all times. The other equally important skill should be the ability to deftly navigate and leverage the political environment of the organization. This job is not for one who does not thrive amongst political heavyweights within the organization. Lastly, the project manager should understand the “schools” of data warehousing. The Inmon graduate will scoff at the Kimball’s Bus architecture with dimensional models and conformed dimensions while the Kimball graduate will never quite take to the 3NF enterprise model of the Corporate Information Factory. One company, having spent a good eighteen months building a dimensional data warehouse, brought in a consultant, indoctrinated to the Inmon approach, to evaluate the state of the warehouse. After spending a few weeks looking at the pure dimensional design without getting into the details of those models, this highly paid consultant served up a scathing report advising the client to pull the plug from the entire effort.

Business Partner(s): These are the identified “super users” in each of the business areas the data warehouse will address. Although the data warehouse team will gather requirements, this business user is ideally an expert in most of the business functions. Examples of such knowledge experts are easy to find in smaller organizations, but much harder in a large global organization. However, having such experts accessible to the team greatly reduces the potential of a gap between what IS develops and what was originally intended.

Core Team of Peers:

The core team has the actual responsibility for the design and development of the data warehouse. In my experience, the most successful data warehouse projects had agile teams where each role had specific responsibilities and none was more important than another. This mindset places emphasis on an environment of mutual respect, a necessary ingredient for any team to operate efficiently. The result is enhanced group dynamics and increased accountability of team members. To be successful with this team of peers, all roles must have ownership of the product’s quality, must act as user advocates, and must understand the business problem they are trying to solve. Unrestricted communication between the Project Manager, Business partner and between the roles in the core team is critical for the success of this mindset and, of the project itself. These are the roles a typical data warehouse / BI project usually sees:

Business Analyst: The business analyst is the equivalent of the product manager in product companies like Oracle or SAP. The BA works with the business stakeholders to understand their needs and goals and translate those into scenarios and quality of service requirements that the development team will use to build the data warehouse and BI infrastructure.

Technical Architect: The architect is responsible for designing the foundations of the data warehouse and BI application. This includes defining the dimensional data model as well as the structure of the BI metadata. The architect’s goal should always be to ensure a solid integrated data architecture and, at the same time, reduce the complexity of the user by dividing the metadata into clean and simple partitions (cubes, business areas, folders, universes etc). This is extremely important because it dictates how the system will be used. The architect should be cognizant of other very important considerations of reliability and maintainability, performance and security standards, and whether the architecture can be evolved easily in the face of changing requirements.

Developers: ETL Developers typically have intimate knowledge of the source systems. They are also expected to have expertise in the ETL tool or programming language. They help build and/or supervise extraction routines and, prepare the warehouse for deployment. BI Developers are responsible for creating canned BI reports and dashboards. They also help users write ad-hoc queries. They are typically experts in the BI tool of choice and have a keen interest and understanding of the business functions.

The other key roles in a successful project are: Testers, whose goal is to discover and communicate problems about the ETL routines or reports that could adversely impact the overall delivery or business value; DBAs to help tune the database for optimal performance and maintain the data warehouse environments; and Trainers, who educate the users on the content and structure of the data, availability and usability of reports and cubes.

Consultants: Most data warehouse / BI projects have a need to fill one or more of the roles I have just described with consultants. However, given that, in real life, data warehousing efforts tend to last a long time, organizations need to employ consultants judiciously. It needs to be clearly understood while a budget for consulting is set up, whether a consultant is being hired for his or her special expertise or whether it is to augment your staff.

In the interest of full disclosure, please visit the About Me page in this blog to learn about my background and the services I provide.

Financial Statement Analysis:

• Trial Balance reports are used to review balances for GL accounts for budgets, actuals and encumbrances for any currency. They are typically run for a date range, a range of accounts
• Balance Sheet detail reports showing balance sheet accounts (assets, liability and equity accounts). Period to period comparisons, month-to-date, quarter-to-date, year-to-date, budget-to-actual comparisons are typical of these reports.
• Cash Flow statements for companies (usually, the first segment of the GL chart of accounts), fiscal periods.
• GL Balances report – this is a detail of all balances in different currencies, for different periods, for different cost centers
• GL detail, or Journal Entry detail reports –to trace the GL balance to all the supporting transactions.
• P & Ls (Profit & Loss Statements) show the revenue, costs, expenses, etc. for different cost centers, fiscal periods. Usually, these are more complicated as further details of the underlying subledgers are usually sought. Revenue analyses by additional dimensions like sales persons are done, some times at the detail of individual billing transactions (or invoices).

Key Financial Ratios: Detail reports are not usually run for Financial ratios and metrics like Current Ratio, Quick Ratio, Debt to Asset, Fixed Asset Turnover, Inventory Turnover (GL), Gross Profit Margin, Net Income, Retained Earnings, Return on Equity, Return on Assets, Asset Turnover, etc. The metrics are calculated for the company and trends over time and comparisons with budgeted numbers are usually of most interest.

Statistical Reports: 

• Headcounts by cost center per company, budget-actual comparisons, and trends over time
• Asset count (e.g. how many laptops do we own?) by cost center, by company, budget-actual comparisons, trended for different time periods or period-period comparisons.

The source of almost all of these metrics is the General Ledger. The source of all detail level GL transactions are Journal Entries. For further analysis, one needs to drill down to the individual Subledger (revenue, cost, expense subledgers). Budget details are also sourced from the GL. Thus, in the Financial data mart, these will be our fact tables:

• General Ledger Balances
• Journal Entry Details 
• Budget details

 Looking at the different analyses above, the key dimensions are apparent: 

• Time (Day, Month, Quarter, Year)
• Fiscal Month (for all aggregate analysis)
• Chart of Accounts
• Set of Books

If you decide to build the subledgers as part of the subject area data marts (e.g. Payables Subledger as part of the payables data mart), then these are dimensions above will need to be used and are the designated conformed dimensions.

A note about Hierarchies:

In Oracle General Ledger, you can define parent-child relationships over the individual segments of the accounting Flexfield. There are no restrictions to the number of levels that you can define. This is an example of the hierarchies for the Company segment of the Accounting Flexfield (actual company name overwritten):

Company 001 can also be defined as the parent of 002 thru 006. All segments can have hierarchies to as many levels as is desired by the business. Hierarchies can also be defined by rollup groups that go across all segments. This is suitable if reporting requirements are known in advance, something that does not happen too often, but is certainly not rare.

Whichever way hierarchies may have been defined, if the business need is to report facts in the form of aggregates and summarize for each of these segments, it is better to have a separate conformed dimension for every segment of the chart of accounts that is important for reporting. In my experience, company, account and cost center are the most common segments that have a need to report at a summary level and hence demand to be treated as distinct dimensions.

Slowly Changing Dimensions: Type 1, 2, or 3?

If you are not familiar with slowly changing dimensions and the different types of SCDs, start with Ralph Kimball’s primer on SCDs.

Accounting Flexfields and hierarchies do not change frequently. Natural account segments change little over time. Cost Center or division hierarchies, however, typically change once every year. This is most common for Sales organizations where regions are redefined and typically change every year. In such cases, the requirement is often to compare fact aggregates (e.g. sales revenue) of the current year’s hierarchy with the previous (or any of the past 5 years’) hierarchy. Requirements of this nature are common and are typical examples of where Type 3 SCDs – the least common of the SCD techniques, are used. This concept is very lucidly explained in this article – “Many Alternate Realities”, again by Kimball.

In the most common scenario, the Chart of Accounts dimension is a non SCD dimension (or Type 1 SCD), i.e. changes are not versioned and the dimension values are overwritten. Company, and Account are also usually in the same category. Cost Center, however, is a conformed dimension having Type 3 SCD technique applied to handle change. Chart of Accounts also are linked to Set of Books (see Part 1 of this series). Set of Books is also a conformed dimension that is used in most financial analysis.

Design of the data mart:

Here is a preliminary design of the Financial Data Mart for GL Balances.
 

To get details of the transactions that make up the balances in General Ledger, we need to drill into the Journal Entries. Here is how that will look:

Budgets will have a similar set of dimensions (not Journal Entry Document or Categories). In Oracle GL, you can create multiple budgets for the same chart of account combination. Hence you will need an extra dimension – Budget (or Budget Version, to be more accurate) to qualify budget amounts for a particular combination of accounting Flexfield segments.

Notice the Journal Entry Document dimension above. This contains various attributes of journal lines like batch name, journal name, description, etc. This dimension plays a key role in “reconstructing” transactions in the data warehouse at the atomic level. It may also be used as a placeholder for several non-additive facts as well. I will explain this very neat concept of Document dimensions in a later post. We have used Document dimensions extensively in my current project.

I have left out several key issues here, most notably that of security and currency conversions. These are subjects that have some unique features and will require much discussion. For now though, I will get on with this basic framework that I have outlined here. In part four of this series, I will explain the data structures in the General Ledger module of Oracle eBusiness suite and will explain how that data can be extracted to this data mart.

If you have found these discussions helpful, please feel free to comment or suggest improvements.

Links to Part 1, Part 2.

The other crucial factors for the success of an EDW from Oracle eBS or, for that matter, any data warehouse / business intelligence effort are:

• Thorough knowledge of the business processes
• Good understanding of reporting and analytic requirements
• A best-practice design
• Good coding practices
• More than adequate testing
• User training and communication

The next obvious question is: How do you know if your DW/BI effort has been successful? David Wells and James Thomann of TDWI describe three “types” of success and their definitions as they relate to data warehousing:

• Economic success – the data warehouse has a positive impact on the bottom line.
• Political success – people like what you’ve done. If the data warehouse isn’t used, it’s obvious that you failed politically.
• Technical success – this is the easiest to accomplish. However, don’t overwhelm your users with too much technology. Success also means that the chosen technologies are appropriate for the task and are applied correctly.

(Source: “Data warehouse quality management,” presented by Dr. James Thomann and David Wells at The Data Warehousing Institute’s Fourth Annual Implementation Conference). More on measuring DW quality.

More often than not, organizations that have sustained data warehousing / business intelligence projects for more than twelve months spend considerable amount of money into this effort. The consequences of a failed DW/BI project are not only in those millions of dollars sunk into the project, it is also in the lost opportunities, wasted resources and unfulfilled potential of the technologies and tools. 

Often, faced with escalating costs and development time, companies put a financial cap. This forces the DW/BI group to narrow its scope and focus to the most immediate and urgent initiatives while losing sight of the initial and long term goals.

To minimize these risks, DW/BI projects within the IT department should learn from a customer focused approach that traditional software companies like Oracle or SAP employ. These product companies talk to customers and research the market to gain a detailed understanding of not just the functional needs and desired product features, but also of the adoption and usage patterns of their product. They keep their product development costs competitive by identifying common customer needs and building the simplest set of features that can cater to the largest group of customers. Additional functionality and options usually come in later releases and in tiered offerings for customers willing to pay a premium those higher value features.

Data warehousing and business intelligence initiatives can succeed by adopting such a user-focused approach. The initial scope of the EDW must be managed well – one that the team can confidently deliver within a short time (ideally, in less than three months). The project team should be able to identify their highest value customers (users) and their critical needs which will then form the core set of requirements or “user stories”. Additional functionality is “released” at a regular schedule. Every release should include an enhanced set of features, additional subject areas, data enhancements and bug fixes to the previous release(s). This process needs to be consistent, measurable and reliable. It keeps the business sponsors and the users continuously interested and engaged in the development of the data warehouse, thereby ensuring political and economic success.

David Chu discusses a practical development approach for data warehouses in this article in DM Review. You can use many tools to manage your agile initiative. I have personally found Microsoft Solution Framework extremely suitable for successful DW / BI projects. The other agile project management software I liked is TargetProcess.

Whether or not you use a tool to manage your DW/BI initiative, it is imperative that project teams secure the involvement of the “customer” at the very beginning and sustain that involvement during all phases of development and testing. The “Business Analyst” or “Scout” should ideally be a step ahead of the development group, engaging with the user to secure and prioritize requirements for the next release even before the current release is delivered. Just as critical is the need to adhere to a definite architecture and be a “pure-player” within that framework. Kimball’s DW Bus Architecture with conforming dimensions and Inmon’s Corporate Information Factory have both achieved tremendous successes, but trying to merge the two or get the best of both architectures is a treacherous endeavor. 

Another practice that I find extremely useful is to get periodic reviews of the data warehouse and BI framework – with the customer (user) to ensure business objectives are still being met and internal assessments within the development team to ensure that best practices and standards continue to be followed in the architecture of the DW, coding of the ETL processes and the design of BI interfaces. These reviews strengthen the relationships with the users and help maintain the strong customer focus throughout the entire project while, at the same time, build a good sustainable standard of development that can help maximize the chances of success.