Kaushal Karnad asked on LinkedIn whether DW/BI projects can be run in an agile manner, as he points out that outputs are mostly reports and extracts and continuous integration would be an issue, having to move data across various DW layers every now and then.
Of course they are! DW/BI projects are, by their nature, to be run in incremental and iterative mode, as the risk associated with a full blown solution is way too high, even more that in traditional software development because of all the non-technical issues related to them. The best approach is to start from the core part of the business, which has the highest priority, and iteratively add new areas and refine what has been released so far, always trying to maximize the value for the customer. And that sounds like agile.
Outputs might be "only" reports and extracts, but what is important is the way you figure out the data and the paths you follow during your analysys: the more you use the system, the more you might feel the need to link apparently unrelated data to gain a deeper understanding of what you're delving into, thus requesting something more (or something different) iteration after iteration: not only new reports, but also new combinations and possibile paths. That sounds like agile too.
Continuous integration is normally about code, not about data, so maybe this is a false problem; anyway, the frequency of updates strongly depends on the real needs of the knowledge workers and often one update a day will suffice.
Showing posts with label data warehousing. Show all posts
Showing posts with label data warehousing. Show all posts
Friday, February 27, 2009
Wednesday, May 28, 2008
A retrospective on seminars
This week I led two seminars for the Data Warehousing course at the Università degli Studi di Milano (see older posts). I found the experience really interesting, because once again, and after so much time, it saw me behind the desk instead sitting of in front of it.
The seminars were about a real (and successful) project which saw me deeply involved for quite an amount of time; the first one dealt with the business aspects, while the second one was more practical and dwelt into the nuts and bolts of the integration of different data sources.
The attendees found the first one far more interesting, 35% of them asserting they had a very high return and received a benefit greater than time invested; another 35% said they received a benefit equal to time invested. The remaining attendees were kind enough to explain me they found it difficult to follow topics which tipically belong to a real world environment armed only with an academic education; the fact that I had to squeeze the work of two years in a couple of hours couldn't but worsen their situation.
The second seminar was only evaluated slightly over the break even, which I really expected as normal given the subject, which is very important but (at least from my point of view) far less appealing. Anyway, to quote Esther Derby and Diana Larsen, "We're happy if most [attendees] feel the meeting was at least a break-even investment ", so I'm quite happy, even if "There's always room for improvement".
I want to thank Professor Silvana Castano for giving me this opportunity and all the attendees, who contributed with their discussions to add value to the meeting - which is, after all, at least in my humble opinion, what seminars are for. Break a leg for your final test, everyone!
The seminars were about a real (and successful) project which saw me deeply involved for quite an amount of time; the first one dealt with the business aspects, while the second one was more practical and dwelt into the nuts and bolts of the integration of different data sources.
The attendees found the first one far more interesting, 35% of them asserting they had a very high return and received a benefit greater than time invested; another 35% said they received a benefit equal to time invested. The remaining attendees were kind enough to explain me they found it difficult to follow topics which tipically belong to a real world environment armed only with an academic education; the fact that I had to squeeze the work of two years in a couple of hours couldn't but worsen their situation.
The second seminar was only evaluated slightly over the break even, which I really expected as normal given the subject, which is very important but (at least from my point of view) far less appealing. Anyway, to quote Esther Derby and Diana Larsen, "We're happy if most [attendees] feel the meeting was at least a break-even investment ", so I'm quite happy, even if "There's always room for improvement".
I want to thank Professor Silvana Castano for giving me this opportunity and all the attendees, who contributed with their discussions to add value to the meeting - which is, after all, at least in my humble opinion, what seminars are for. Break a leg for your final test, everyone!
Friday, May 9, 2008
Seminars update
It is confirmed that I will be leading two seminars for the Università degli Studi di Milano as an add-on for the data warehousing course. Dates and arguments follow.
- May 26 - Designing a Data Warehouse for an intercompany information system
- May 27 - Data Integration: a case study
Wednesday, April 16, 2008
Speech Engagement Schedule
As a follow up to this article, it is confirmed that I'll be leading a seminar on data warehousing for the Università degli Studi di Milano on May 26/27, 2008 (the date is yet to be confirmed, but it is fairly set).
The seminar will be about a real project I led, and it will be conceptually organized in two parts: the first part will deal with the case study in general, while the second one will focus on the integration of different data sources. Depending on the needs of the audience, the seminar will be held in one or two sessions.
The seminar will be about a real project I led, and it will be conceptually organized in two parts: the first part will deal with the case study in general, while the second one will focus on the integration of different data sources. Depending on the needs of the audience, the seminar will be held in one or two sessions.
Thursday, February 7, 2008
Star schema
OLAP (On-Line Analytical Processing) tools are among the most common front end systems for data warehouses; they allow dynamical and multidimensional analysis to be performed against a huge amount of records in order to produce a small set of data which can be used as a dashboard for business process management by the so-called the "knowledge workers".
There are two widespread approaches to OLAP implementation: ROLAP (Relational OLAP) and MOLAP (Multidimensional OLAP). Ok, there is also an hybrid solution, called... you guessed, HOLAP (Hybrid OLAP).
Why should people use the relational model to implement a multidimensional model? There are many reasons, the most important being the diffusion of advanced RDBMSs and the expertise of IT people. Moreover, ROLAP systems don't have a "sparse data" problem, thus being far more scalable than MOLAP implementations. Unluckily, the relational - and bidimensional - model, in which we find attributes, relations and integrity constraints, has a reduced expressivity when it comes to describe the multidimensional model, in which we find facts, measures, attributes, dimensions and hierarchies. That's why we have to find a workaround, which leads us to the (notorious) star schema.
The star schema consists of one (or more) fact table(s), which represents facts, referencing many dimension tables, which represent the dimensions of analysis. Fact tables typically have a lot of columns, and newbies almost always smell the rat of a very bad use of the relational model where they should see a very good compromise instead.
One of the reasons behind the star schema is the very poor performance shown by RDBMS when they have to aggregate a huge amount of records belonging to many different tables, thus involving many expensive join operations: denormalization can then improve performance at the cost of the increased disk space required. Another way to improve performance is redundancy: you materialize derived tables (views) based on the most used aggregations to speed up typical analysis. In addition, ROLAP implementations often use surrogate keys, another feature that make newbies shrug.
The star schema can have some variations, as the snowflake schema, obtained decomposing one of more dimension tables eliminating transitive functional dependencies contained in the tables. Dimension tables which keys are imported in the fact table are called primary while the others are called... why, secondary, what else?
There are two widespread approaches to OLAP implementation: ROLAP (Relational OLAP) and MOLAP (Multidimensional OLAP). Ok, there is also an hybrid solution, called... you guessed, HOLAP (Hybrid OLAP).
Why should people use the relational model to implement a multidimensional model? There are many reasons, the most important being the diffusion of advanced RDBMSs and the expertise of IT people. Moreover, ROLAP systems don't have a "sparse data" problem, thus being far more scalable than MOLAP implementations. Unluckily, the relational - and bidimensional - model, in which we find attributes, relations and integrity constraints, has a reduced expressivity when it comes to describe the multidimensional model, in which we find facts, measures, attributes, dimensions and hierarchies. That's why we have to find a workaround, which leads us to the (notorious) star schema.
The star schema consists of one (or more) fact table(s), which represents facts, referencing many dimension tables, which represent the dimensions of analysis. Fact tables typically have a lot of columns, and newbies almost always smell the rat of a very bad use of the relational model where they should see a very good compromise instead.
One of the reasons behind the star schema is the very poor performance shown by RDBMS when they have to aggregate a huge amount of records belonging to many different tables, thus involving many expensive join operations: denormalization can then improve performance at the cost of the increased disk space required. Another way to improve performance is redundancy: you materialize derived tables (views) based on the most used aggregations to speed up typical analysis. In addition, ROLAP implementations often use surrogate keys, another feature that make newbies shrug.
The star schema can have some variations, as the snowflake schema, obtained decomposing one of more dimension tables eliminating transitive functional dependencies contained in the tables. Dimension tables which keys are imported in the fact table are called primary while the others are called... why, secondary, what else?
Wednesday, February 6, 2008
Star schema?
Today we started coaching two customers (two IT people) on our data warehousing tool. They have a lot of expertise in their software and a vast knowledge of their operational domain, not to mention the intensive use of SQL, so all went quite smooth until we talked about the structure of the tables in the data mart. Or should I say... the structure of THE table (the fact table), as the dimension tables are quite simple to understand.
For a newbie it is quite a shock to see a star schema: all that redundancy... many unused columns... SO MANY columns... referring to unrelated data... Actually there are a lot of reasons behind this schema, and I shall talk about them... sooner or later ;-)
For a newbie it is quite a shock to see a star schema: all that redundancy... many unused columns... SO MANY columns... referring to unrelated data... Actually there are a lot of reasons behind this schema, and I shall talk about them... sooner or later ;-)
Wednesday, January 23, 2008
Some things start out small
Now it's more than a rumor... I have a speech engagement schedule, even if there's only one event... which should be in May. I'll be presenting a seminar on data warehousing at the Università degli Studi di Milano, as an add-on to the data warehousing course. Some news should also appear on the official site of the DICO department of the University; stay tuned for more informations. The seminar should, more or less, be based on my thesis; that means I should be quite prepared, so I hope I won't have too many groceries thrown at me. And if I have... given the prices of groceries, at least I hope they'll be fresh.The picture is taken from the beautiful "Dinosaur" Disney movie, from which I've also taken the title of this post. The full quote goes "Some things start out big, and some things start out small, very small. But sometimes the smallest thing can make the biggest changes of all."
Subscribe to:
Posts (Atom)
