(responding to Rich, Vasil)
(Rich)
I'm mostly lurking here out of interest, rather than as an
active practitioner, but I do do data warehousing.
Looks like a good story, thanks for sharing it with us.
For what it's worth, we have not had any great success with
the equivalent of BDUF.
;-)
We originally built our warehouse to solve a specific problem (in
or case trying to track the federal government's welfare reform
reporting requirements). We had an ulterior motive here in that
it also allowed us to get our core data collected on the warehouse.
After that was successful, we got hubris and decided to begin
work on an "Enterprise Wide Data Warehouse." The goal is
worthy, even valuable, but our attempt to build it as one big
project fell flat.
There were two reasons for this. First the project is just too big.
That's a common problem. It's almost always easier to break
big projects into smaller ones - even when they need to be
run in parallel and closely coordinated. If they don't, then it's
even easier. I was about to say 'better' rather than 'easier',
but there are many criteria that could come into play.
We have hundreds (>3,000) programs, dozens of data sources
and several thousand users. Trying to get all of this documented
and architected took forever.
Understandable.
In the meantime our users and fellow developers had all of their
other work to do (It didn't help that we were doing this in 1999)
and we could not get the participation we needed to collect the
information. It didn't help either that they weren't all all that sure
what the value of all this work was.
You definitely want to have the goal, cost and value up front.
However, in general terms; relate the cost and value to
business objectives, risks etc. not to dollars at this stage
(i.e. rough out a business case). The vision of the project
grows from this core.
After about a year of this we bagged it. Instead we created a
loose plan of how we wanted all of it to fit together when we were
done. While original effort was helpful here, we could probably
have done this a year earlier.
Right. Before you have this 'vision' you're like a boat without
a rudder. A "loose plan of how we wanted all of it to fit together"
sounds like a good starting point, once you know what you're doing,
for whom, and why.
Then we attacked specific points of pain in our user group. We
got a number of them together and figured out how we could sofve
a big chunk of their problems with the data we had already
collected. We worked with them, and trained them to get their
own information out.
Maximise the value of what you've already got - also gives folk
an idea of what you might be able to do for them. Sounds good.
This helped build enthusiasm for the project and we started
getting pushed to add more data to solve other problems.
When we did we made an effort to try to fit it in with what we
already have. We more or less succeeded in that. Where the
integration is a problem we've worked to fix it.
Okay, going at the big picture in a piecemeal fashion will
expose you to the need for more change than going at it BDUF,
so you need to make change cheap. OTOH, if you go BDUF
and get anything wrong, the cost of change is horrendous.
If you managed to achieve results, sounds like you made the
right choice. How's the quality of code? Is it degenerating, or
are you managing to keep it fresh? What about the structure
of the data? Is that holding up well? If both of these look
good, congratulations, award yourselves a gold star ;-)
As a result we are on our way toward getting our Enterprise Wide
Data Warehouse, but we're doing it in fits and starts solving
specific problems as we go.
That sounds like a good way to go. Do things when they
get high enough priority; do something else if that is more
important. Beware of keeping switching contexts, though.
You probably only want to consider switching 'projects'
at the end of an iteration - emergencies excepted.
First, it's doable. (Always a good thing).
Sometimes you can't know that at the start, but if so, you need to
find out as soon as possible. Here you're already familiar with
the problem, so should never have that to deal with.
Second, we've built a supportive environment. Data warehousing
is expensive; you need to keep you're uses productive. By
building it in stages to resolve real problems we've been delivering
value all along.
Apart from that first false start, of course. Yes, that's good.
Finally, we're slowing gaining the value of overall integration in
that we are now able to answer questions and do analysis on
issues that no one has ever thought to ask before this. The synergy
of data warehousing is very real.
Emergent properties, no less!
Is it agile? I have no idea.
If the code and data quality is holding up, I reckon you get
your label. Doesn't mean you should rest thinking you've
got there; I'm still learning, and I don't see why I should let
others stop ;-)
(Vasil)
Does anyone know or have experience of how I could
apply agile principles within a data warehousing environment?
Any help would be appreciated as the current process I am
working within has a heavy emphasis on a waterfall approach.
... However, there's an Agile group just being set up in the
Thames Valley area ( don't have contact details, maybe
Brian Hanly of eXoftware could help you there? Or Rachel
Davies of Amarinda, who's also in the area ) ...
Hah! I checked for contact information - it seems you need
to contact yourself ;-) Oh well.
Paul Oldfield
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
www.aptprocess.com
any opinions expressed herein are not necessarily those of
Mentors of Cally or the Appropriate Process Movement
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
For more information about AM, visit the Agile Modeling Home Page at www.agilemodeling.com
--^----------------------------------------------------------------
This email was sent to: gcma-***@gmane.org
EASY UNSUBSCRIBE click here: http://topica.com/u/?bUrKDA.bWnbtk.Z2NtYS1h
Or send an email to: agilemodeling-***@topica.com
TOPICA - Start your own email discussion group. FREE!
http://www.topica.com/partner/tag02/create/index2.html
--^----------------------------------------------------------------