Ideas on Enterprise Information Systems Development

This blog is devoted to ideas on Enterprise Information Systems (EIS) development. It focuses on Lean Thinking, Agile Methods, and Free/Open Source Software, as means of improving EIS development and evolution, under a more practical than academical view. You may find here a lot of "thinking aloud" material, sometimes without scientific treatment... don't worry, this is a blog!
Every post is marked with at least one of Product or Process labels, meaning that they are related to execution techniques (programming and testing) or management techniques (planning and monitoring), respectively.

Wednesday, December 15, 2010

Enterprise Information Systems Patterns - Part II

The project in Github reached an initial implementation, which I separated in a branch named sketch, given that it is mixing configuration and instantiation. The master branch now is in accordance with the proposal, which I continue to discuss below.

Remember that the core idea is to use the concepts as Lego parts. This mean avoiding using subclasses in general, but masking the abstract concepts through configuration. After configuration, I need to implement specific functionalities for the concepts.  These functionalities will be implemented in two places:
a) At instances of concrete concepts, by defining specific behavior for
different contexts (paths).
b) At path objects, in the form of coordination code, which will make the path''s
movements collaborate to realize a business process.

In that way, we are going to have a two-phased development/customization process:
a) Configuration: defines descriptors, which represent concrete uses of the
abstract concepts. Descriptors list the types used to transform the abstract
concepts into concrete ones. Configuration is done through a Domain Specific
Language (DSL), having each concept a template text to be used for its
configuration. In the future, we expect to define a proper grammar for this DSL.
b) Implementation: uses descriptors to make the concrete concepts instantiable
and implement the specific code related to their concrete use. Each concept has
a callable object with a proper name, which is defined during the implementation
of user stories.

Thus, in a first moment, a domain specialist will configure concrete concepts
using the DSL. Configurations are reused during the implementation, when
user stories instantiate and define the specific behavior of the concrete
concepts.

Right now, we have implemented Resource and Node, as well as the Maskable superclass, which will serve all concepts. I can say that the project is in the configuration phase yet, meaning that we haven't implemented any code for "using" the concepts in some business process.

Regarding the DSL used for configuration, we have to investigate its relation with our BLDD proposal. Remember that this proposal is used for implementing business processes, while the current DSL is used for configuring objects, which occurs before implementation of the processes. Maybe an execution DSL which uses the same terms used for configuration will grow from this investigation.

Thursday, December 9, 2010

Assorted Thoughts on Agilism X Traditionalism - Part V

Am I against UP, PMBoK, UML, and Certification?
(a post that summarizes some opinions of mine)

No, I am against the way these techniques were transformed by some people. I call it the Plague of 40-hours-courses: people hear about some "new" and fashionable technique, and rush to attend to some expensive one-week course on this technique. They don't try to learn from other sources, at most they buy some book to glance sometimes. Usually, they buy expensive tools that "make your process adherent to something-fashionable"*, believing that if they use the tool correctly, everything will work fine.

If you don't practice, if you don't try things, if you don't drink from different fountains, you won't have a nice process, because organizations differs from each other, and there isn't something like the Lapis Philosophorum, which will turn any development process into gold.

The typical managers' answer to the advice above is " we don't have time for experimentation", and then they buy packaged courses and tools, which won't work well many times, making managers go to the next phase of the project for process improvement, which is either blaming the team, either blaming the technique. Others start doing bad things, using the new technique to justify them.

Managers shouldn't waste time controlling small things, but studying better ways of making things self-controlled, on making their teams work more productively and with more quality, or, in other words, becoming the facilitator of good practices. Things like checking if everyone is filling their daily working time in some "project management" tool does not aggregates value to clients. In fact, doesn't aggregate value to you and your team. What all this has to do with the theme of this post? The answer is that if managers used their times to understand and adapt techniques to their process, instead of acting as controllers, they would use these techniques in better ways.

The way people transform good ideas into waste-generation machines is amazing! So, people transformed
(i) UP phases into waterfall cycle.
(ii) PMBoK into a heavy and expensive annotation effort.
(iii) Modeling into the main software development activity.
(iv) Certification into a reason for not evolving processes.

Regarding these techniques, I think that
(i) UP is a nice list of "things to worry about during product construction". But you don't have to use all of its workflows exactly how they are presented. UP is a framework, therefore, it exists to be adapted.
(ii) Similar to UP, PMBoK is a nice list of "things to worry about your process management", but you don't need to use all of its documents. PMBoK is a framework, therefore, it exists to be adapted.
(iii) Modeling is for understanding the domain area and also for communication. The users' demands are the focus, and users need running code, not models. Check out Martin Fowler's UML as Sketch principles.
(iv) Certification is a moment to review and document your practices, but you should never forget of continuous improvement. More important than saying to others that you work well is actually working well! IMHO, certification would be a process of checking the quality of products and of asking the clients about your service. Snapshots of the documents used in your process don't provide any real guarantee of quality in the long, or even the medium, term.

*In fact, worse than the Plague of 40-hours-courses is the Plague-of-Tools-That-Make-Your-Process-Adherent-to-Some-Fashionable-Technique. Some years ago I heard a happy colleague saying that "could finally understand object orientation", after attending a course on an UML case tool...

Sunday, December 5, 2010

Assorted Thoughts on Agilism X Traditionalism - Part IV

Some days ago I bumped again into an "old" internal discussion by email on the Agile 2009's talk "Lets stop calling it 'agile' ", which summary I copy below:
"
Agile development has grown a lot since its rebeleous 2001 start. In fact, it has grown to be the mainstream way of developing software. The time has come to drop the word "agile." Agile development is just modern practices in software development. There is no need to explicitly mark practices as Agile. There is no need anymore for opposing camps. Keeping the word Agile and things like "the Agile conference" is holding the development of modern SW development practices back. This session will be in debate form to discuss the above mentioned motion.
"
Some can find this radical. When I first read it one year ago, I also thought the same. However, reading it now I can clear understand the proposal, and it made me remember of my way through software processes.

During my undergraduation course I had contact with Yourdon's Structured Analysis as well as Object Oriented Analysis & Design, by that time (1990-1994), the first was mainstream and the second a new proposal which needed to be improved. However, the second one attracted me more, given that I wanted to use Object Oriented concepts the most, since my first contacts with C++ in 1991 I was already convinced that OO was a better way of programming.

First Experiences
Outside the classroom, my first professional contact with "analysis" occurred in 1993, when I took part of a Management Information System (MIS) development in Clipper and Structured Analysis and Design. On those days of waterfall cycles, someone did the analysis and design and I implemented the system. As expected, after deploying it, users asked for changes almost every week. Firstly I  implemented them and once a week updated the modeling artifacts, for maintenance purposes . Yes, first I changed the programs, because my users were eager to get the software running with the changes, and them, when had time, updated the models. Looking back I see how natural is the idea of using models only to understand the problem, and updating them when needed.

In early 1994 I inverted positions, acting as the analyst for another MIS, using Structured Analysis and a classical waterfall cycle. The implementation was successful until at least its deployment, later in the year. After that, I lost contact with it.

In the second semester of 1994 I entered a bigger Enterprise Information System project, with circa 15 developers and two consulting companies working on it. It was very uncommon environment, using IDEF artifacts for modeling, C++ for programming, and Sybase. Transformation from functional models to C++ methods were done during interesting CRC Cards sessions, and class diagrams were derived from Entity-Relationship Diagrams. Object-relational mapping was done according a set of very basic rules, created internally by the team. The only problem we had was the fact that some process models were prepared almost one year before we started to touch them. However, we had a very good business analyst (he was an Industrial Engineer) who re-checked all business process with the clients before they enter the programming phase. Therefore I can say that the process was interesting, however, the "waterfall way" of doing things created a lot of rework.

Before I forget, effort estimating in all cases was done by "expert opinion",  people didn't believe in Cocomo's use of constants and in Function Points counting - which needs a lot of things already defined to have a chance of being "accurate".  I could only remember my Software Engineering classes and started to see them as something closer to Alice in Wonderland than to an Engineering discipline.

Enters Object Modeling Technique (OMT)
However, it was only in 1995 that I had the first chance of using professionally an OO method in the whole process, specifically, I used the OMT method. OMT was interesting because it used a mixture of class and state diagrams with Data Flow Diagrams (DFD's) to describe its Functional Model. OMT was a product of its time: OO languages were becoming mainstream, but Structured Analysis was still the mainstream process paradigm.

Our Environment in 1995
In this year I moved back to my hometown, aiming to take part of the creation of the first big and public university in the region. We were a two men team, in partial time, working at the Math Laboratory - from where the Computer Science Laboratory would emerge years after. The mission was to develop a library management system to be used by the three campus' libraries. The university was new and many, many information systems were necessary. Library Management Systems where really expensive by that time, thus the university's president asked us to create the system.


Our Process in 1995
We decided to use a mixture of Prototyping Life Cycle with OMT's artifacts to build the system, since we had very few knowledge in the problem area (libraries). For more complex operations, we would use CRC cards to define how classes would interact. We decided to use Delphi 1.0, which allowed us to create nice GUI as well as use Pascal, instead of Visual Basic (which I hated). Moreover, my colleague was developing an object-relational mapping library in Pascal for his Master's Thesis, which gave us a cleaner and powerful way of doing Object-Relational Mapping (ORM). We resisted the temptation to use Delphi's components for direct access to the database for the sake of really using object orientation for coding business logic.


Firstly we drafted a list of high level requirements - separated in modules, and then we asked the librarian to prioritize them. Then we started to follow a process that we built on top of our experiences in previous systems:
1) Get the next priority.
2) Detail its requirements.
3) Create a prototypical GUI and a draft database to reflect the requirements.
4) Re-check requirements during GUI presentation to the user.
5) Model stuff in OMT.
6) When appropriate, use CRC cards to define classes' responsibilities and collaborators.
7) Write code ignoring the database*.
8) Write ORM and database code.
9) Present the implementation to the user.
10) If the user was happy with the results:
a) we deployed the new module and went back to 1)
else
b) we looped back to 5) if modeling was needed, 6) if changes in classes were likely to occur, even without new modeling, or 7) if only changes in algorithms were needed.


* We used an ancestor technique of what is called today "test doubles". In our case, collections of objects with typical values were created "by hand", so that we could test the algorithms. When we put the algorithms to work properly, we then implemented the persistence code, and finally we took way these "fake objects". Sometimes I developed algorithms and my colleague put things to work with the database, since he was the ORM library developer. We didn't want to waste time with persistence problems while we were trying to solve users' problems. Persistence was a pure technical question, not a main development concern.

Effort estimation was done before each of these "proto"-iterations, by experience, because I kept the same opinion regarding effort metrics. The project was really successful, we both left the university in the end of 1996, however it was used for years, until the university bought a new system, with external support from a development company.

I used this process more or less unchanged until 2000, when I had access to my first UML Case Tool, and started reading about the Unified Process and Eriksson-Penker's business modeling extensions for UML. I knew UML since 1997, however, by that time, and until 1999, I had only one project, which used OMT already, and I had no reasons to change from it to UML. I used UP variations until 2006/2007, sometimes without the expected output. And all times not fully implemented, since I thought many things were too bureaucratic to be used.


The main thing I learned from these days was that you must mix techniques and even create new ones to make the development process work for your environment, and not the opposite. Hardly you will find a process that fits exactly to your environment, team, culture, and problem. I've seen a lot of people eager to follow some guru or certification, and when things go wrong they blame themselves for not being able to do exactly "what must be done." They should blame themselves for not being able to adapt the process to their environment.


Another thing I can clear identify now is that some recent Agile techniques are modern versions of old programming tricks, those kind of hints shared during the coffee break, and that never appeared in the "software engineering" books.


Going back to the "drop the word agile"
I think my intimate history in dealing with development process is the same of a lot of others developers. In early 1990's, there were a lot of different Object Oriented notations and processes, all seeking to solve the same problems, in slightly different ways. Hybrid methods such as OMT (and its DFDs) were told to be the ideal solution for a world in transition. And for me they seemed to be.

Then came UML and UP, both bringing some standardization to the software development world. Finally, Model Driven Engineering (MDE) and Model Driven Development (MDD) could realize their full potential. And they did, given that I think their potential has finished. At least, the potential to be innovative in research terms, and to answer to constant changes in practice terms.

I agree with the phrase "Agile development is just modern practices in software development." Yes, one should try to list a single innovation that came from the MDD world in the last few years. Every "new" MDD technique that I saw recently, either was repetition, either was "extreme cases", hardly useful.

Many people still resist to Test Driven Development (TDD), for instance, because they can't understand how tests can be used for design. There is a common misunderstanding that design is related only to blueprints, thus, design = modeling in software development. What about Stress Analysis, mostly based on Math and Physics, instead of drawings... so design is not only about blueprints, and, actually, if you can rely on Math to do the design, it is more likely that your design will work. As I said before, coding is closer to Math than modeling.

For a developer to understand how TDD can drive design, he/she must practice it. You can draft a model to understand the problem in the beginning, however, given that you are going to interact with tests - which are code - and the code that answer to them, you are going to evolve the design as the need for testing evolves. An tests evolve as the implementation of requirements evolves. The problem is that many decision makers are those that "don't program anymore", so will "never" be convinced, unless some programmer show to them how things work.

There is also an economic and cultural reason for people not recognizing agilism as the "modern" software engineering: agile techniques mostly were - and still are - nurtured by individuals and communities of developers, in opposition to big consultancy companies. Because of this, Agile techniques will take longer to enter the EIS world. People will repeat to the exhaustion "Why big companies like A or B don't use a lot of Agile if it is so good?", and will answer to themselves "because it doesn't scale!". Isn't because you don't need expensive tools and related consultancy to adopt them? Isn't because of culture? Isn't because a lot of people who stopped programming after becoming project managers are afraid of seeing part of their daily activities labeled "waste", thus endangering their jobs?

I witnessed similar reactions in the early 1990's regarding Object Orientation, and if the "legacy guys" were right, I would be saying stuff like "Structured Analysis rules!".

So, drop the word Agile!

Wednesday, December 1, 2010

Enterprise Information Systems Patterns - Part I

This post aims to start the discussion on design patterns for EIS. We have just started a Github project, which right now is just a rough sketch, however, it already shows some basic ideas, implemented in Python with BDD (specloud, should-dsl, and Lettuce). By checking the features (*.feature files) you can understand the patterns requirements, and by checking the specs (*_spec.py files) you can understand how the code implements them.

We decided to start by revisiting ERP5 concepts, separating them from ERP5's underlying platform, Zope, and, in some cases, even reinterpreting them. The concepts are the following:

Concept 1: Resource
A resource is anything that is used for production. It can be a material, money,
machine time, or even human skills.
Concept 2: Node
Is something that transforms resources. For instance, a machine, a factory, a
bank account.
Concept 3: Movement is a movement of any Resource from one Node to another.
Concept 4: Path*
It describes how Movements are used to implement business processes.

ERP5 also has the concept of Item, an instance of a Resource. Until this point, we
decided not to use this, otherwise it would be necessary to create classes for
instancing the other three abstract concepts.

The core idea is to use the concepts as Lego parts. This mean not using
subclasses in general, but masking the abstract concepts through
configuration. For instance, a Movement is firstly configured as a concrete
movement, such as transferring goods from supplier to customer. After that, it
can be instantiated. Of course, at some point extra coding is necessary, and we believe that methods for implementing specific algorithms will emerge
on the coordinator element, the Path instances.

We therefore define two new "patterns": Mask, which means to define an abstract concept and turn it into a concrete concept by a proper configuration, and Coordinator, which is an object (also maskable) that coordinates others to realize a given business process. Maybe these patterns already exist with others names and/or similar purposes, however, we will check this more carefully as the project goes forward (we are really iterative & incremental).

The motivation for this project is that people like to say that frameworks are like Lego, but when you need to subclass their concepts, instead of using them as is, I think this is not the "Lego way" of doing things. We only use masks to provide useful labels for users, but the intention is not to extend the original concepts.  When I mask a Movement as a Sale, I am only setting its mask attribute to 'sale'.

The only concept that will be extended is Path (nothing is perfect!), since it will be used to implement the specific algorithms necessary to appropriately integrate movements - realizing the business process.

We probably won't supply something really useful for programming, because we are not considering platform issues, therefore, if you need a ready to use ERP framework in Python, check ERP5. The intention is to discuss how to build highly configurable frameworks, instead of proving a real framework. After we get the basic configuration machinery done, we are going to use BLDD to implement didactic cases: from the business process modeling to coding and testing.

(The Change Log maps this series of posts to commits in GitHub)