Project monitoring in business intelligence
Business Intelligence (BI) is a technological process that provides companies with strategic and operational decision-making tools. Data analysis is the basis for this.
The data is extracted, modified, and loaded using a specific tool: the ETL (Extract, Transform, Load). The data is then stored in a data warehouse to obtain intelligence presented via restitution tools. This is what is known as the decision-making information chain.
Based on the criteria given by Ralph Kimball, considered to be the reference in the field, on the life cycle of the BI project, it is possible to adapt methods such as the V cycle or even agility to BI. The issue here is twofold: it concerns the choice of method and its adaptation to the BI framework.
1. Use of the V-cycle in Business Intelligence
The V-cycle consists of a top-down and bottom-up phase and is based on end-to-end development, which can take several months. It avoids returns in case of anomalies are encountered because the production of the application is completely delegated until the final delivery.
- Security: each phase starts only when the previous phase has been closed.
- Preparation: the study of the specifications is carried out before any development
- Control: a verification system is set up in front of each specification phase.
- Change management: the V-cycle does not allow for new decisions within a cycle. Changes should be integrated between two cycles.
- flexibility: delay in the specification phase will mean less time for testing, jeopardizing the quality of the delivered product.
Dependency on one or more external data sources is a distinguishing feature of BI projects. It can have important consequences, as it will be difficult to advance in the phases and to propose delivery dates when a critical element goes outside the scope of the project team.
1. Launch of the project
You start the reflection by looking at the final product, and you work your way up. We look at the decision support required and then see how to achieve it.
During this stage, documentation is written indicating what needs to be done during the project.
2. User interviews:
This stage is crucial because it enables the project to be tailored to the user’s needs and to ensure the expected business value. In fact, in business intelligence, the users are the only ones to define the project and its needs.
To help in the decision-making process, we must succeed in thinking like the user, being functional by using the information he gives us. The technical aspect will therefore not be addressed until the detailed specifications stage.
3. Functional specifications
At this level, the “what?” of the project is answered. In decision-making, it consists of:
- List the indicators with their functional definition, the list of axes, and their hierarchy in detail. This is called the data dictionary.
- Define the reports to be delivered
- List data sources
- Define the functional architecture. This corresponds to the major blocks of the system, in order to begin to go into detail while keeping a view of the business needs and problems.
- Set up the operational CDM (Conceptual Data Model): this is often neglected. It must be validated by the user. It corresponds to the existing functional structure of the activity under consideration but will have to be denormalized for the next step.
- Implementation of the decision-making MCD. It provides a cross-functional view to help with management and decision-making. It is integrated into the ETL for feeding. We generally seek to have simple queries that respond directly to user needs.
For example, if we want to know the number of trains from one country to another on a given date, the operational MCD may look like the following:
Once denormalized, we can obtain a decisional MCD as follows. Here, the “Station” entity appears twice to meet the decisional need, whereas operationally it should only appear once since it represents a single concept.
During this stage, the “How?” of the project is answered. The focus is on the ergonomics of the application and the final rendering, which in business intelligence is usually of two types:
- Reporting: dashboards (2 dimensions)
- Cube (n dimensions)
The idea here is to quickly set up browser masks with locally predefined reports to show users what will be delivered visually. We need to show a dynamic image of future deliverables. To do this, scenarios are drafted and played out in front of the client, allowing him to validate the interest of the project. This stage concludes with the drafting of a mock-up report on the remarks made during the user presentations.
5. Detailed specifications
Following on from the mock-up stage, the “How?” of the project is answered here. The technical aspect is addressed. For information purposes, the list of elements making up detailed specifications in decision support is as follows:
- In order to prepare the implementation in an RDBMS, the PMD (Physical Data Model) of the Data Warehouse must be foreseen because this model is close to the constraints of database software. Tables, fields, and identifiers (primary and foreign keys) are defined. We also study the performance (indexes, denormalization, etc.).
- Technical architecture file
- Definitions: Indicators, Axes (dimensions), Reports, Rights management
- Datawarehouse feed procedure: incremental, cancel/replace
- Procedure for recovery in case of failure
- Discharge management procedure
- Graphical user interface (GUI) specifications
This phase corresponds to the implementation of the elements as defined in the detailed specifications. In the case of business intelligence, it is most often the manipulation of tools and the writing of SQL queries. Development is divided into two teams:
- One deal with ETL for the data warehouse
- The other takes care of the return.
Unit tests are carried out on both sides and documented, assuming that the user knows the tool at this level.
7. Integration and Acceptance
Integration consists of pooling the modules developed separately: ETL and restitution. Then comes the acceptance phase, during which the tool must be tested, failures must be identified and corrected, and all this must be recorded in an acceptance book. There may be several levels of acceptance depending on the usefulness and the need to which the tool responds.
8. Mistakes to avoid in change management
Change management takes place in parallel with the project cycle. It helps to ensure that users accept the changes.
In the case of the V-cycle, it is especially important to avoid the tunnel effect, when a project is behind schedule and, above all, offers little prospect for its progress and completion. To do this, it is necessary to ensure that the client is given visibility throughout the project and to take into account any constraints that may be added.
As regards external contributors to the project, it is necessary to have good sponsors before starting and not hesitate to challenge what the publishers of the software used to promise.
As with any project, one should only commit to what can be achieved. Nor should naive expectations be raised through decision-making, such as “the decision maker will decide better” or “turnover will increase”.
With regard to the use of data, a common mistake is to propose an operational rather than a decisional CDM and, in the end, not meet the user’s needs. Favoring numbers while forgetting textual data, being technically oriented rather than use user-oriented, and putting data in the warehouse quickly so that it is available, are common mistakes in BI that must be avoided.
2. Application of agility in a decision support project
1) Presentation of agile methods
The use of agile methods on a BI project is a continuous process. A complete BI solution will not be implemented in one shot, but rather through a succession of scripts of equal size and duration to allow the project to adapt to user needs. Agile methods propose to involve the customer more during the construction of the product. It is an iterative process. The requirement is not fixed and can evolve throughout the project, adapting to the constraints and progressive understanding of the project.
Agile m, therefore, moves away from the traditional V-model in which communication between developers and business users is not a priority. In addition, developers focus more on the data and technology than on the need to find answers to important questions.
In concrete terms, Agile BI links the Technical and the Functional, instead of focusing on the complete implementation of one of the three stages of BI project management (ETL / DWH / Reporting). The latter will be split into several functionalities, themselves split according to the BI project management stages. Thus, when each function is completed, it will be integrated with the previous one.
The idea is to give weight to the technical path through the more traditional functional project path. Management is first horizontal and then vertical
2) Course of action
a) The concept
The success of the BI project requires clear and realistic objectives, combined with a study phase of the existing situation. This strategic mastery will enable us to anticipate and manage deadlines.
It is also important at this stage to identify the types of information that decision-makers will need to improve their management through data. Precise indicators must be determined bearing in mind the strategic framework of the project: a global or industry-specific KPI (Key Performance Indicator) will not necessarily be relevant.
During this stage, the stakeholders will take part in the process for the first time. We find here the first point of Kimball since it will be the occasion to put forward the added values of agility for the business (better follow-up, transparency on the evolution, adaptation to a new need). This will be an opportunity to define the different priorities:
- Train the project actors in the fundamentals of the agile approach;
- Identify the main needs and requirements of the company;
- Find out what data sources are available;
- Understand the channels for disseminating information: reports, dashboards
- Prioritize key operational requirements and needs within time and budget constraints;
- Choosing which business intelligence software to use
The action plan should also specify the hierarchy and tasks of each person in the follow-up and implementation of the BI project.
c) Construction of the Project
A functional system must be provided that meets the changing needs of stakeholders. This means going through this stage continuously and at fixed intervals, usually one to three weeks. Several methods are used depending on their compatibility with the project and the company’s organization.
During construction, it is important to :
- Developing collaborative reports;
- Use just-in-time (JIT) modeling: identify a problem to be solved, call in colleagues, and explore the problem together, then resume work as normal.
- Finalize the tests ;
- Finalize documentation ;
- Release the pilot project to a small sub-group: the project is tested in production on a small portion of its scope in order to get a vision of what Go Live will be and to identify the last gaps;
- Train end-users ;
- Train production staff ;
- Deploy in production
b) The production
The Production stage is the final deployment phase of the project (often called Go Live). During this stage, it will be about:
- Operate and support the system, dashboards, and reports;
- Identify defects and improvements. All these changes should start at the construction stage and continue through to production.
Once in production, agility will allow the product to evolve iteratively.
3. To conclude:
Ralph Kimball, in his book The Data Warehouse Lifecycle Toolkit (1998), gave 3 concepts for the lifecycle of the data warehouse and therefore of the BI project. These concepts can be put into perspective with the two types of methodologies mentioned in this article:
|Kimball concepts||V-cycle||Agile Methods|
|Focus on adding business value to the whole company||OK: by defining from the beginning of the project what is expected and by staying close to the users||OK: By defining at the beginning of the project what is important for the business and adapting the tools to the different needs|
|Offer data across dimensions (time, geography, department, occupation, etc.)||OK: This aspect will be implemented during the functional specification phase where the data to be integrated into the DWH (Facts/Indicators and Dimensions) are modeled||(OK): This aspect is independent of agility. In each iteration, we focus on one (or more) fact tables and their associated dimensions.|
|Develop an iterative solution, deliver understandable increments||KO: V-cycle is an end-to-end development||OK: In agile, deliverables are bricks of functionality: we cut them into portions that the user can understand. In BI, we deliver a single business process, for example, which can be further broken down according to its dimensions or data sources.|
HeadMind Partners Data consultancy service supports its customers in their digital transformation around Business Intelligence:
- Choice of the BI project monitoring method according to the company’s environment and production capacities
- Implementation of the decision support tool according to the decision information chain.
Website: Eiden, Florian “Business Intelligence Project Management – Agile or V-Cycle Methods? BI is winning you over! .2012. [Accessed 05/08/20]. Available at: https://fleid.net/tag/cycle-en-v/
Website: Eiden, Florian ” BI project management: keep your users close to you! BI is winning you over .2012. [Accessed 05/08/20]. Available at: https://fleid.net/2011/12/20/gestion-de-projet-decisionnel-gardez-vos-utilisateurs-proches-de-vous/
Website: Boyer, Clément ” Lecycle en V ” SupInfo International University .2017. [Accessed 05/08/20]. Available at: https://www.supinfo.com/articles/single/6278-cycle-v
Course: RIGAUD, Lionel. BI Project Management. Business Intelligence Option, CY TECH (EISTI), 2012
Book: Kimball, Ralph “The Data Warehouse Lifecycle Toolkit” 1998
Deltil, E., & Pereira, G. “BI – Business Intelligence: Presentation” Université de Marne La Vallée. 2017. Available at: https://www-igm.univ-mlv.fr/~dr/XPOSE2006/DELTIL_PEREIRA/presentation.html#need
Cartelis. “Comprehensive ETL Software Comparison: Cloud vs. On-Premise vs. Open Source” Cartelis. 2018. Available at: https://www.cartelis.com/blog/comparatif-logiciels-etl/
Supinfo School of Computer Science. “Understanding the steps of a BI process” Supinfo. 2017. Available at: https://www.supinfo.com/articles/single/3548-comprendre-etapes-processus-bi
Fernandez, A. “The GIMSI method” Piloter.org. 2018. Available at: https://www.piloter.org/mesurer/methode/fondamentaux_gimsi.htm
Fernandez, A. “The new managerial dashboards: the whole decision-making project”. 2011.
Wikiversity. “Strategic project management: Decision-making project” Wikiversity. Available at: https://fr.wikiversity.org/wiki/Gestion_de_projet_strat%C3%A9gique/Projet_d%C3%A9cisionnel
Lau, S. & Sabatier, J. “Business Intelligence: The place of BI and the management of BI projects in large French organisations. Business Intelligence: Place de la BI et pilotage des projets décisionnels dans les grandes organisations françaises” CIGREF. 2009. Available at: https://www.celge.fr/wp-content/uploads/2015/09/Business_Intelligence_CIGREF_2009.pdf
Sodifrance. “Stages and concepts of a BI project” Sodifrance. 2009. Available at: https://blog.sodifrance.fr/les-etapes-et-notions-dun-projet-bi-2/
Deltil, E., & Pereira, G. “BI – Business Intelligence: The steps of the process” Université de Marne La Vallée. 2017. Available at: https://www-igm.univ-mlv.fr/~dr/XPOSE2006/DELTIL_PEREIRA/process.html#collection
Lecomte, S. “BI project: 4 steps to success” Alphalyr. 2019. Available at: https://alphalyr.fr/blog/projet-bi-etapes-pour-reussir/
Cartelis. “Data Warehouse Architecture – Traditional vs. Cloud Approaches” Cartelis. 2018. Available at: https://www.cartelis.com/blog/architecture-data-warehouse/
Tehreem, N. “Data warehouse concepts: Kimball vs. Inmon approach” Astera. 2020. Available at: https://www.astera.com/fr/type/blog/data-warehouse-concepts/