Ideas on Enterprise Information Systems Development

This blog is devoted to ideas on Enterprise Information Systems (EIS) development. It focuses on Lean Thinking, Agile Methods, and Free/Open Source Software, as means of improving EIS development and evolution, under a more practical than academical view. You may find here a lot of "thinking aloud" material, sometimes without scientific treatment... don't worry, this is a blog!
Every post is marked with at least one of Product or Process labels, meaning that they are related to execution techniques (programming and testing) or management techniques (planning and monitoring), respectively.

Thursday, August 18, 2011

Enterprise Information Systems Patterns - Part XII

Step by Step Programming using EIS Patterns
This post will describe, step by step and using the example "bank system", how to program using the framework and Fluidity + Extreme Fluidity.

Step -1
You must know the following techniques and tools:
a) BDD using Lettuce.
b) The Decorator Design Pattern.
c) Should-dsl, to define expectations in both acceptance and unit tests.
d) Python Unittest Library.
e) How to use Python Decorators (@).

Step 0
Understand how and why things are done in the framework:
a) Start at Part IV, the previous parts focus on discussions and experimentations which are not used in the framework anymore.
b) Why Decorators are used to extend classes, explained in Part V.
c) Why should-dsl is also used, besides in tests, to define contracts between objects, explained in Part VI.
d) How things described in parts IV to VI are implemented in general is described in Part VII -  if you feel it too abstract, I suggest going to Part IX.
e) Part VIII only describes the status of the project by that time, you can ignore it, but it is necessary to accompany the change log to become aware of important changes in the code.
f) Part IX describes conceptually how to program using the framework. Part X describes in more detail the use of Fluidity, and Part XI explains the evolution of the concepts. This post does the same of Part IX, but showing the code instead.

Reading the posts will give you an idea of the modus facendi of the framework, however, it is necessary to start programming... probably it is necessary to go back and forth from code to these posts. The following steps are a kind of guide, but in no way it is meant to describe a complete development process, this is one thing that you developer should infer. Some details may be omitted to avoid too long descriptions.

*It is strongly recommended that you check the unitary tests to understand how the framework core classes work in detail*

Right now we are using BDD because we don't have a nice BLDD tool to work with. Therefore you are supposed to describe the business process in BDD terms, by interpreting a template workflow, and then incrementally define the features' steps.

The following steps were developed on top of a specific example, however, they can be generalized.

Step 1
Based on the bank system example, let's suppose that the code is part of a bigger system which have already core concepts such as Employee and Bank Account.
First the process is defined in Fluidity, them, incrementally, the scenarios are defined and finally their steps.  
Fluidity:
    state('requested')
    transition(from_='requested', event='create_loan_request', to='request_created')
    state('request_created')
Scenario:
Scenario Outline: Individual Customer asks for loan
    Given I am a registered Credit Analyst
    And an individual customer with account number <account number> asks for  a personal loan
    And the loan request is of <desired value>
    When I confirm the loan request
    Then a new loan request with the <account number> and <desired value> is created
    And the new loan request is associated to the Credit Analyst
Steps:
@step(u'Given I am a registered Credit Analyst')
def given_i_am_a_registered_credit_analyst(step):
     world.a_person = Person()
     an_employee_decorator = EmployeeDecorator()
     an_employee_decorator.decorate(world.a_person)
     world.credit_analyst = CreditAnalystDecorator('09876-5')
     world.credit_analyst.decorate(world.a_person)

Step 2:
By analyzing the code above it is identified the necessity of developing a credit analyst business decorator. You will do only the basic necessary to the first step pass: create a skeleton of CreditAnalystDecorator, and go back to next step afterwards.
At this point it is already important to define the Rule of Association for this decorator, which is "to be a credit analyst, someone must be a bank employee first". Before writing the rule code, a test for checking if the rule is being enforced and if the decoration is working properly must be prepared:
    def it_decorates_a_person(self):
        #should fail
        (self.a_credit_analyst_decorator.decorate, self.a_person) |should| throw(AssociationError)
        #should work
        an_employee_decorator = EmployeeDecorator()
        an_employee_decorator.decorate(self.a_person)
        self.a_credit_analyst_decorator.decorate(self.a_person)
        self.a_credit_analyst_decorator.decorated |should| be(self.a_person)
        self.a_credit_analyst_decorator.decorated |should| have(2).decorators

Now the rule's code can be written:
    @rule('association')
    def rule_should_contain_employee_decorator(self, decorated):
        ''' Decorated object should be already decorated by Employee '''
        decorated |should| be_decorated_by(EmployeeDecorator)

This rule will assert that only if the person is an employee then he/she can be a credit analyst, and it is implemented as a method of the credit analyst decorator class. The be_decorated_by matcher was created specifically for this framework.

Attention: as of versions 0.7.3 of the core and 0.1.3 of the example, a singleton Rule Manager object started to be used, in order to promote reuse of rules. This object is composed by a "rules" object, which in the core is called Core Rules. Checking the tests of the example it is possible to see that other rule base is used, in the case a subclass of the core rules, called Bank System Rule Base. It is important to note that some tests use the core rules, simply because they don't need to use rules specific to the bank system. In other words, if you need a new rule, it must be created into your rule base object - making it hard-coded, however it is envisioned a way of loading them from a file or other configuration object, maybe using an even higher DSL to define them.


Step 3:
Now that the basic credit analyst decorator is defined and tested, we go back to the acceptance level, in this example, we will jump straight to the "Then" step because the others are straightforward. At this point, the business process starts to be assembled.
@step(u'Then a new loan request with the (.+) and (.+) is created')
def then_a_new_loan_request_with_the_account_number_and_desired_value_is_created(step, account_number, desired_value):
     (...)
    Creates the business process object:
    world.an_individual_credit_operation = Process('Individual Customer Credit Operation')
    Defines the process' source and destination:
    world.an_individual_credit_operation.set_source(world.the_company)
    world.an_individual_credit_operation.set_destination(world.a_client)
    Configures the process using a template state machine:
    template = LoanProcess()
    Uses xFluidity's to get a configuration based on the template:
    configurator = StateMachineConfigurator(template)
    configurator.configure(world.an_individual_credit_operation)
    At this point, the process object (an_individual_credit_operation) has absorbed the template state machine
    Now it is time to configure the first movement - loan request creation
    The object the_movement will be a proxy for CreditAnalystDecorator
.create_loan_request. This movement will be a Transformation, having credit_analyst.decorated (the associated person) as both source and destination, and an_individual_credit_operation.create_loan_request as the activity - that's an important point: create_loan_request is a method, symmetric to a workflow transition, which was injected into the process object by xFluidity.
    the_movement = world.an_individual_credit_operation.configure_activity_logger(world.credit_analyst.decorated, world.credit_analyst.decorated, world.an_individual_credit_operation.create_loan_request, CreditAnalystDecorator.create_loan_request)
    After this, the collection of movements of the process should contain the recently configured movement (found by the activity's name)
    world.an_individual_credit_operation.movements |should| contain(the_movement.activity.__name__)


Step 4:
At this point the CreditAnalystDecorator class has no create_loan_request method, thus it is time to go back to the unit level and write a test to this specific requirement:
    def it_creates_a_loan_request(self):
        an_employee_decorator = EmployeeDecorator()
        an_employee_decorator.decorate(self.a_person)
        self.a_credit_analyst_decorator.decorate(self.a_person)
        self.a_credit_analyst_decorator.create_loan_request(self.an_account, 10000) #self.an_account was created in the setUp()
        self.a_person.input_area |should| contain('1234567-8')

Now it is implemented as an @operation:

    @operation(category='business')
    def create_loan_request(self, account, value):
        ''' creates a loan request '''
        loan_request = LoanRequest(account, value, self)
        #Places the loan_request in the node's input area
        self.decorated.input_area[loan_request.account.number] = loan_request

Note that a LoanRequest class must be defined, which drive us to the next step.

Step 5:
The first thing to do is to create a unit test for LoanRequest and then create the class itself, which are trivial tasks. After running all the necessary tests, we can go back to the acceptance level, given that the loan request class now exists and it is operating properly.

Step 6:
    Time to run the loan request creation and store the execution context:
    the_movement.context = world.an_individual_credit_operation.run_activity(the_movement, world.credit_analyst, world.account, desired_value)
    Since the create loan request transition was fired, the process should be on state 'request_created'
    world.an_individual_credit_operation.current_state() |should| equal_to('request_created') 


Step 7:   
Now it is time to check the 'And' clause of this step, which is:

@step(u'And the new loan request is associated to the Credit Analyst')
def and_the_new_loan_request_is_associated_to_the_credit_analyst(step):
    #... this association was done during loan creation, just checking
    world.a_person.input_area[world.account.number].analyst |should| be(world.credit_analyst)

This step was easy, since the way the loan request creation was implemented already places it on the decorated (the Person) processing_area.


The next step would be go to the next process transition and implement it using a sequence similar to the one presented above.


In summary, developing in the framework involves:
1) Describing a user story guided by a business process template.
2) Developing the code for the story steps, which will identify new responsibilities, to be implemented into the decorators and resources.
3) Developing the code for the decorators, including defining their association rules and @operations.
4) Developing the code for resources, including defining their association rules.
5) Detailing the steps: use movements to wrap and associate @operations to each workflow activity, providing full control through the workflow engine while logging the execution context.

The use of the process template and the association of decorators' methods to business process activities provides the adherence of the application logic to the process' logic. Currently, BDD is in use, therefore the linking to the process logic is done manually. However, it is envisioned the creation of a tool to generate the steps skeletons in a way adherent to the process logic.

Thursday, August 11, 2011

Enterprise Information Systems Patterns - Part XI

Evolving the interpretation of the concepts
As the framework evolves, and we play with it through the examples, new interpretations of the concepts appear. In fact, after playing with Fluidity, we realized that Transformation and Transportation were not necessarily implemented as classes, but categories of movements instead, given that they behave in the same way - it is only a question of explicitly saying if the work item is transformed into a Node (a transformation) or simply moved from one node to another (a transportation).

The history behind this is that, when developing the example, it was noticed that even a "simple" resource movement would need to be logged, as well as it could be associated to, or even generate, other resource(s). When we realized this, it was clear that transportations would potentially need the same machinery used by transformations, hence, they share a common behavior, which should be implemented into Movement class. Therefore, they were transformed into categories - although the Category class is not in use yet. Figure 1 shows how the framework's ontology is interpreted now.

Figure 1: EIS Patterns Ontology

Another important point is that the relationship "is_a" between Operation and Resource does not represent a classic object oriented inheritance relationship. In other words, Operation is not a class, instead, it is a Python Decorator which is applied to Person and Machine methods to mark them as Business Operations. Therefore, when it is said that "a Node performs an Operation", it means that some of the Node's method is decorated as an operation - or, this node is able of performing this operation. Finally, the relationship "operates_on" between Operation and Resource means that operations are always transforming or transporting resources, through movements which encapsulates them (the operations).

In summary:
a) An operation is an abstract representation of a production operation.
b) A node's method is decorated by an operation to inform that the original abstract concept is now represented concretely by the method.
c) Inside a business process, a movement is used to encapsulate and log the execution context of the concrete operation defined in (b).
d) Running a process consists of executing and logging the movements defined in (c).

Part IX of this series explains in less abstract terms how to deal with operations and movements. Part X explains how to do this using Fluidity and Extreme Fluidity. As expected, in order to understand how the whole thing is implemented, it is necessary to check the EIS Patterns Examples project, in special, how BDD steps are implemented.

Thursday, June 9, 2011

Enterprise Information Systems Patterns - Part X

EIS Patterns, Business Processes, and Workflow Engines
Every EIS is based on business processes, and it can be said that workflows are the computational realization of these processes. Therefore it is necessary to provide some workflow engine to EIS Patterns.

Basically, there are two types of workflow engines:
a) Associated to an application development platform: such as the ones found in Web application development environments, like Zope, or associated to databases, such as Oracle, or as part of Groupware environments. In this case, applications are developed "inside" the environment, using the engine's API.
b) Implemented as Libraries: using some programming language, a library of objects representing the workflow features is implemented, such as the State Machine as Python Decorators, or Ruby's AASM. In this case, you must use the libraries' objects to implement your engine.

Enters Fluidity & Extreme Fluidity
Fluidity was developed as a type (b) workflow engine, however, Extreme Fluidity turns it into a third type of workflow engine:
c) Inoculated & Expelled: using the dynamic nature of the underlying programming language, provide a way of making objects workflow-aware, and symmetrically, a way of turning these objects back to their initial structure when desired.
Why this? Because type (a) engines forces you to use a given environment to develop your applications, while type (b) forces you to use specific objects to implement workflows, most of times creating a mixed code with application specific and workflow specific statements.
With Extreme Fluidity (or xFluidity for short) it is possible to insert the code necessary to make your workflow run inside your business objects. In other words, they become workflow-aware objects, while keeping your programming style, standards, and patterns. In order to understand how this happens, let's check how Fluidity can be used when xFluidity is associated to it (the complete source code can be found here).
First, it is necessary to declare a template State Machine:

class LoanProcess(StateMachine):
     state('requested')
     state('request_created')
     state('request_analyzed')
     state('refusal_letter_sent')
     state('loan_created')
     state('value_transfered')
     initial_state = 'requested'
     transition(from_='requested', event='create_loan_request', to='request_created')
     transition(from_='request_created', event='analyst_select_request', to='request_analyzed')
     transition(from_='request_analyzed', event='loan_refused', to='refusal_letter_sent')
     transition(from_='request_analyzed', event='loan_accepted', to='loan_created')
     transition(from_='loan_created', event='time_to_transfer_value', to='value_transfered')

Using xFluidity, the template is injected in another object - in this case a Process:

    def setUp(self):
        self.process = Process()
        template = LoanProcess()
        configurator = StateMachineConfigurator(template)
        configurator.configure(self.process)


Magically, the process object starts to respond to the state machine's events:

    def it_makes_the_process_respond_to_the_example_state_machine_events(self):
        self.process |should| respond_to('create_loan_request')
        self.process |should| respond_to('analyst_select_request')
        self.process |should| respond_to('loan_refused')
        self.process |should| respond_to('loan_accepted')
        self.process |should| respond_to('time_to_transfer_value')

As expected, the engine avoids the firing of transitions at wrong situations:

    def it_runs_the_example_acceptance_path(self):
        self.process.create_loan_request()
        self.process.current_state() |should| equal_to('request_created')
# the line below means that if the process tries to fire loan_refused
# when at 'request_created' state, an exception is generated
        self.process.loan_refused |should| throw(InvalidTransition)
        self.process.analyst_select_request()
        self.process.current_state() |should| equal_to('request_analyzed')
        #loan accepted
        self.process.loan_accepted()
        self.process.current_state() |should| equal_to('loan_created')
        self.process.time_to_transfer_value()
        self.process.current_state() |should| equal_to('value_transfered')
In that way, it is possible to use a workflow engine in your own development environment and using your own models. Fluidity also provides guards and state actions, not shown here for the sake of simplicity.

A Note on Workflow Notations
Fluidity is a state-based engine, which on those days of fancy business process notations may appear a bit old-fashioned. However, state machines represent a well known and easy to understand notation. EIS Patterns' Process and Movements objects are being developed in a way that they don't need to know which notation is in use, at least at the unit level - meaning that some code specific to a given notation may be used to "glue" elements at the application level.

Conclusion
Currently Fluidity and xFluidity are works in progress, including the expelling capabilities and some details on implementation of guards. When these features become fully functional, a Petri Net based engine will be developed. A clear limitation of Fluidity is that it only works for Python objects, but the necessity of developing in a given programming language is a limitation of all workflow engines, although some provide XML-like APIs.

Sunday, May 29, 2011

Enterprise Information Systems Patterns - Part IX

How to use EIS Patterns?
When eispatterns started to be developed, the original goal was to build a set of basic concepts that could be configured to represent various business entities. However, this seemed to be very hard to accomplish, since as new behavior appears, new code is necessary, making the economy of using masks to represent new entities being not enough to justify the use of the framework.
Therefore, the framework's focus shifted to on how to represent real world entities that dynamically acquire new responsibilities. Then, a new path was followed, by defining:
-Essential abstract concepts in the framework: Node, Resource, and Movement.
-The roles for each of the abstract concepts: Node: active, responsible for executing production operations; Resource: passive, responsible for storing  production resources data; and Movements: coordinative, responsible for managing the nodes while they work on resources.
-How to extend each of the concepts: Node: through Decorator Pattern; Resource: through subclassing; Movements: through configuration (better explained here).

Programming in EIS Patterns
In the project repository a super-simplified "bank system" can be found, with the intention of demonstrating how the framework can and should be used. The framework is process-centric, meaning that the development should evolve around the underlying business processes, by following these steps:

1) Represent the business process using a Process object. Ideally, this object should "absorb" the attributes and methods necessary to run as a workflow. For instance, if a state machine is used to represent business processes, the process object should have states (attributes) and transitions (methods), as well respond to specific events. Moreover, if it is decided to represent the same process using other notation, such as BPMN, this object must be transformed accordingly. We implemented Fluidity, a state machine for Python, which responds to these demands*. Process objects use simpler Movement objects to connect business operations to business processes and log their execution.

2) Define business decorators to adorn Node objects. Nodes are persons or machines which gain new responsibilities through business decorators, which in turn manipulate Nodes' input, processing and output areas. Business decorators make Nodes become the active part of the business process. Decorators must implement a decorate() method, where rules of association are defined using should-dsl. These rules ensure that a given object can be adorned by the decorator.

3) Each process activity (using a state machine: a transition or state action) must be realized by a business decorator method. These methods are called (business) operations, can be categorized, and are encapsulated by Movements**, which in turn are linked to the Process object. In that way, business decorators' methods take the responsibility of making the process work***. The same template can be used to configure different processes, using different operations to implement them.

4) Define the operations by decorating them with @operation. These operations should be able to ask for the transference of resources from/to the different nodes' areas using node's transfer() method. @operations implement the logic for each business process activity.

5) Define WorkItems, which are, in general, data bags or passive entities****. The reason for that is that in a real-world, physical production system, humans and machines transform or transport the work items using their abilities (@operations). A report is passive, being written by a human; a truck hood is passively spot-welded by a robot in a factory; a motor is passively transported from one manufacturing cell to another.

These steps are not necessarily executed in order, nor the business process is necessarily fully defined beforehand. It is envisioned a development process guided by the incrementally testing/building of the business processes. The use of templates is also a form of defining the acceptance criteria for the tests.

It is important to note that every class and every method must have a docstring, so that keyword searches can be performed by the Documenter module, which will be (soon) extended by Natural Language Processing facilities, so that requirements can be parsed and used to try to find candidate reusable elements in a given library of business decorators and work items. Complimentary, the Rule Checker is a module that runs the association rules of the business decorators found by the search, so that users can verify which of them can be used for adorning a given object. A new module will be developed to find and run  acceptance tests, so that the "live system" can be used to check its fitness to a given set of requirements.

*Note 1: In comparison with workflow engines and workflow libraries, Fluidity (in conjunction with Extreme Fluidity) is innovative, because while engines force the development inside their environment and libraries force the use of their objects to implement the process, Fluidity allows the definition of template machines that can be absorbed by any Python object. In other words, you can still use your own classes, your own naming convention, your own programming style, and your own architecture, in your own environment. The "state-based behavior" is inoculated into your objects dynamically. Moreover, if you don't want to use Fluidity anymore, your objects can expel the state machine at any time.

**Note 2: Movements are of the type Transformation - which transforms a given resource in a node's processing area, and Transportations - which transports a given resource from a node to another. These concepts became conceptual classifications of movements, only Movement objects do the job.

***Note 3: Implementing business process is a two step task: first a movement is configured to log an @operation execution, and then this operation is executed.
Using a state machine notation, the property movement.activity is set to a transition or a state action, and the  property movement.activity_associated_method is set to @operations. It would be easier if transitions and actions would be directly associated to @operations, however, Movements are necessary to store execution information (parameters, result, date and time) as well as to indicate the source and destination nodes. Source and destination nodes allow the mapping of the process to physical movements in the production plant. After configuration, a movement can be executed. These steps are implemented into Process objects, because they coordinate movements. This layered method improves reuse and configuration of processes and @operations.

**** Note 4: If workitems are physically aggregated to form a final product, this set of workitems must be promoted into a Kit, which can be "active" in some contexts.

Thursday, April 14, 2011

Enterprise Information Systems Patterns - Part VIII

The current status of the project

The code in commits of April 9 presents the first version of Rule Checker, a class able of finding Business Decorators' rules implemented in should-dsl and run them against the object to be decorated. In April 11 the initial version of Documenter was committed. Documenter simply returns the documentation of Resources and Business Decorators. Rule Checker and Documenter form the basis for the following configuration process:

a) Requirement input phase
    1) User enters the keywords that represent the main concepts of the requirement.
    In the future:
    2) User enters the requirement's text.
    3) A Natural Language processor will identify the main concepts of the requirements.
b) Concepts search phase
    1) By simply matching keywords, the framework suggests candidate business decorators and resources which can be reused to configure the new business process.
    In the future:
    2) Based on documentation and rules of association, an ontology is built to enhance searching of candidate decorators and resources.
c) Concept compliance testing
    1) To better understand how candidate decorators work, run automated tests, so that the user can see the decorator in action.
    2) To check if the decorator can be used to decorate a selected Node object  (Person or Machine), test which decorators can be applied to it by checking (running) association rules.
d) Configure Business Process
    1) Using a domain specific language (DSL), describe the logic of the business process and generate the appropriate code for the business process.
    2) With the information obtained in (c), configure the business process. If new and/or changed functionality is necessary, enter the process of developing it, using the partially configured business process as a basis for acceptance tests. In that way the business process is constructed incrementally. At this point, the use of BLDD is envisioned.

The core idea of the process above is to provide ways of creating new systems in a dynamic and flexible way, through configuration . By dynamic I mean allowing the user to manipulate live objects which can demonstrate their capabilities - going far beyond documentation reading; by flexible I mean the use of Decorator Pattern to create new business entities on the fly. In the cases where changes are needed, the use of BLDD can provide a safer and dynamic way of extending the framework with new or improved business decorators and resources.

The code necessary to implement this process is in the following state:
-a.1: create a nice GUI, planned to medium term (end of the year).
-a.2 & a.3: work in progress based on NLTK, planned to medium term.
-b.1: done.
-b.2: planned to medium term.
-c.1: planned to short term.
-c.2: done.
-d.1: dsl for state diagrams is a work in progress.
-d.2: planned BLDD use for the medium term, full BLDD to long term.

Monday, March 28, 2011

Why Open Source for EIS?

Short Answer
It is safer. And cheaper. And auditable...

A bit longer answer
I could stop here because every IT professional heard this at least once in life, but companies still rely a lot on Windows in both client and server sides. Today I read from a big Brazilian newspaper that USA's Department of Defense estimates that 55,000 new computer viruses appear a day. Guess which operational system these viruses aim for?

If you are not convinced, I will summarize some figures I found here (as of June 2010):
• Windows:
-1 to 1.2 billion MS-Windows virus signatures
-Only 1 million signatures are checked by an antivirus => 0.1% can be scanned
• Others:
-270 Apple Mac (OS-9 or older) viruses
-No known Apple Mac OS-X viruses as of June 2010
-60 Linux viruses (of older distributions)
-No know Linux viruses as of June 2010
• And more:
-55,000 new Microsoft Windows malware per day (as of March 2010)
-Compare to 22,000 daily average in 2008 and 40,000 daily average in 2009.

Why people still use Windows and MS-Office in companies? The answer relies on legacy. Legacy technologies, and a terrible asset found in many enterprises: IT people with legacy minds.

Sorry for being so direct and a bit rude, but I challenge anyone to prove, unless recent heavy investments justifies, that Windows and Office are not replaceable by Linux and Libre Office on most enterprise areas. Libre Office does 95% of what Office does, and the other 5% probably you will find few users in your organization who really need it. Some other softwares, in special the graphical tools, are not Linux-friendly in general, however, even for companies with design bureaus - people that uses CAD and such -  again you will find many users that don't need Windows. And still, why use MS-Office?

The IT Legacy Minds
I think that with the figures above, it is very unlikely to justify "scientifically" the use of Windows in most organizations, even when we talk about previous investments. Think of the money spent on Windows, Office, and anti-viruses licenses. Sum up the cost of repairing infected machines, the loss of information, the resulting productivity losses, the private data disclosure, and you see that the cost of training people to use Linux and Open Office is much, much lower.

A real problem is with legacy systems, written to run in fat Windows clients. In the case of business systems, if they are more than 10 years old, it is justifiable, if not, it is strange that your developers or your software supplier haven't developed them for the Web, given that one of the things that were clear after the Y2K problem was that fat-client information systems were much more expensive to maintain. In other words, unless some clear necessity of using fat clients justifies, IT people made a wrong decision.

In some specific areas, such as industrial automation, fat clients are really necessary, however, even in these cases, you can develop in Linux. I myself took part of a team which developed, deployed, and supported, from 2005 to 2009, an industrial application made in LabView for Linux. In fact, we were one of the firsts to put in production PXI-SCXI hardware running Linux in the world, and we did it for an offshore oil producing plant - a quite severe environment.

Therefore, I feel free to conclude that the main barrier is really the lack of IT professionals to deal with Open Source solutions, and this problem is caused by  the number of IT professionals that simply don't want to study new things. It seems paradoxical, however, in my opinion, IT people are quite conservative!

There are also personal interests involved: if the organization adopts an IT structure that is safer and bring much less security problems, maybe some people will loose their jobs (you can find a fable on this type of employee behavior here).

Conclusion
Finally, I would like to mention a post from the Agile Scot blog: Department of Defense (DoD) is #1 using Open Source for Government. As the title suggests, the DoD uses a lot of Open Source, thus, even if your organization is bigger than the USA's DoD, do you think you have more tentatives of invading your systems than them?

Tuesday, March 22, 2011

Enterprise Information Systems Patterns - Part VII

Understanding how concepts are implemented
In Part V of this series, I used an UML Class Diagram to show the relationships among the framework's concepts. Although this diagram can help understanding these relationships, being it a class diagram, it also "says" that all concepts are "classes", which is not true. A better way for describing these relationships is using an ontology, as shown in Figure 1.
Figure 1: Ontology representing the relationships among framework's concepts

Operations, since they are immaterial resources, need to be realized by some active entity, in this case, by Nodes. Therefore, they are implemented as "functions" (in fact "methods"), instead of instances of a given class. Although in Python functions are also objects, one cannot define them as objects of a given business class. However, it is necessary to typify Operations somehow so that we can identify among some object methods which represent business operations. The solution for this is using Python Decorators, which are not exactly the same thing as the Decorator Pattern used in this framework. A Python decorator is a specific change to the language syntax that allows to more conveniently alter functions, methods, and classes.

Thus, it is possible to create a @operation(a_category) pythonic decorator which will store into method objects an attribute named category, with value a_category. In that way it is possible to mark methods as of Operation type, which is important to query a given class for its business skills. For instance, in a bank information system, it is possible to query a Credit_Analyst class to check which business operations objects of this type can perform. Or the contrary, using a keyword search discover, identify which methods of which classes may perform a given business operation. Moreover, I can create an ontology and navigate through it to check relationships among resources, nodes, and movements. With this ontology I can even suggest to a business analyst specific configurations for the system, given appropriate search terms.

Processes are sets of transformations and transportations, which in turn are used to encapsulate operations. Encapsulate means to forward calls to the appropriate Node subclasses methods - in fact to their Business Decorators methods - and log all information related to these calls, such as date and time, parameters and associated objects. Thus, Process objects implement the logic which coordinates how Nodes work. In other words, Processes implement workflows, which means, in turn, that the framework is process centric, or workflow centric.

Decorating, Decorating, and Subclassing
To make things more clear, let's call "decorators" the extension to Python syntax, and "business decorators", the classes created to decorate the framework concepts, as explained in Parts IV and V. Said that, let's check how the framework is implemented and extended:
a) Resources:
-Operations: methods, implemented into business decorators and marked with @operation decorator (yes, decorators for decorators).
-Work Items (formerly known as Materials): "classical" objects, for each new type of material, a new subclass of material is created.
-Kits: "classical" objects, for each new type of kit, a new subclass of kit is created. Given that kits can be composed by Operations, which are in fact methods, this composition is implemented through a list of references to these methods.
b) Nodes:
-All three subclasses are extend through business decorators. For instance, a person can be decorated by Developer (business) decorator to represent a software developer, or a Credit Analyst (business) decorator to represent a bank's credit analyst. No subclasses used. They can have methods related to @operations, to @rules of association (check Part VI), as well as methods representing internal machinery, in this last case, typically private.
c) Movements**:
-Transformations and Transportations are used as is, no subclassing, no pythonic decoration, no business decoration. They are used to build processes and store information related to operations calls, by encapsulating @operations during their configuration process.
-Processes are extended by mimicking workflow templates (current work), or by using workflow engines to make them work. A way of implementing processes is using Business Language Driven Development implemented through State Machine using decorators*.

For the reasons presented in Parts IV and V, subclassing is avoided at maximum - it is openly suggested only in the case of Work Items (Materials), which are considered passive "data bag" classes.

*Post-publishing note #1: we decided to develop our own State Machine code, and make it independent of this framework, check the Fluidity project.

**Post-publishing note #2: Transformations and Transportations are not classes anymore, as explained in Part XI.

Friday, March 11, 2011

Enterprise Information Systems Patterns - Part VI

A DSL for Rules of Association
The use of rules of association may facilitate configuration: by querying them it is possible to check, for a given concept, which responsibilities can be assigned to it. The definition of a Domain Specific Language (DSL) for creating rules of association would be interesting, because it would allow the construction of more readable rules. In fact, a branch of should-dsl can be used. Should-DSL is a DSL for assertions, which are commonly used by coding techniques based on design by contract.

In fact, should-dsl's Predicate Matchers and some of the already available "ordinary" matchers such as respond_to,  which checks if an object has a given attribute or method, and be_instance_of, which verifies if an object is of a given type, can be used for formatting rules. For instance, let's suppose we have a decorator that is supposed to work only with persons:

class A_person_decorator:

    def __init__(decorated):
        ...
        #checks if decorated is compliant to the rules
        self.check_rules_of_association(decorated)
        #if it is compliant, sets a pointer to it
        self.decorated = decorated
        ...

    def check_rules_of_association(decorated):
        try:
            #uses be_instance_of matcher to check if decorated is compliant
            decorated |should| be_instance_of(Person)
        except:
            raise RuleOfAssociation('Person instance expected, instead %s passed' %  type(decorated))     
     
    ...

In the example above, type checking is necessary given the nature of Python language, however, the focus is in creating more complex rules related to the business rules.

Another interesting possibility is rule querying:
(i) For a given decorator, what are its rules of association (list its rules)?
(ii) For a given decorated, which decorators it can use (list which responsibilities a given class can assume)?
Query results could be presented in both human and machine readable ways. In other words, it would be some mechanism for querying rules, for both automatically checking associable objects in a object base, as well as informing people what is needed for configuring a concept to receive a given type of decoration.

A funny situation is that should-dsl matchers are used in unit tests to check the correct functioning of the rules... written in should-dsl.

The next post will describe some more details that decorators need to implement.

Wednesday, March 9, 2011

Enterprise Information Systems Patterns - Part V

Decorators X Subclasses
In the previous post, I redefined the level of abstraction for the framework, and now it is time to discuss a bit about Decorators. In order to facilitate the communication, a simple UML Class Diagram is provided by Figure 1, representing the structure after this redefinition.

Figure 1: UML Class Diagram of the new structure

In Figure 1, each of the (now) three abstract concepts (resource, node, movement) has two "opposite" subclasses (operation X material, person X machine, transformation X transportation) and one aggregator subclass (kit, organization, process). It is important to note that  aggregators are Composite objects.

By (re)checking the Decorator pattern documentation, both on Wikipedia and on the classic book from Gamma and colleagues, we can find that:
- While subclassing adds behavior to all instances of the original class, decorating can provide new behavior, at runtime, for individual objects. At runtime means that decoration is a "pay-as-you-go" approach to adding responsibilities.
- Using decorators allows mix-and-matching of responsibilities.
- Decorator classes are free to add operations for specific functionalities.
- Using decorators facilitates system configuration, however, typically, it is necessary to deal with lots of small objects.

Therefore, by using decorators it is possible to, during a business process realization, create an object, associate and/or dissociate different responsibilities to it - in accordance to the process logic, and log all this. In that way, I have two main benefits:
i) The same object, with the same identifier, is used during the whole business process, there is no need for creating different objects of different classes.
ii) Given (i), auditing is facilitated, since it is not necessary to follow different objects, instead, the decoration of the same object is logged. Moreover, it is possible to follow a single object during all its life-cycle, including through different business process: after an object is created and validated - meaning that it reflects a real-world business entity - it will keep its identity forever.

Summarizing...
Thus, the benefits of using Decorators are:
i) More dynamic and flexible enterprise systems, through the use of configuration and pay-as-you-go features.
ii) Easier auditing, given the fact that objects keep their class and identification while get new responsibilities.

An example
To better understand this, I will use a simple example: a teacher & researcher career. Let's suppose in our institution we have two kinds of teachers, the 20-hour and the 40-hour. While the first one is supposed only to teach, the second is also a researcher, and therefore holds more responsibilities. Given that there is vacancy, a teacher can change from one to another category. Since both types are "teachers", the ordinary object oriented solution would be to create a basic class named Teacher, which would hold the common features of the teaching career, and two subclasses, named 20-hour Teacher and 40-hour Teacher. With this architecture, I can see no simple solution than creating objects of the different classes and copying attributes back and forth every time someone changed his/her option of working 20 or 40 hours.

Moreover, imagine that teachers can also be assigned to administrative positions, such as department dean or university president, with a lot of new responsibilities associated, while still teaching. Keeping track of all these assignments and withdraws of responsibilities would be complex and error-prone. Also, I like to think that being a dean is getting a new responsibility, instead of becoming a different type of employee, in special when we think that this is a temporary assignment.

Now imagine that we have Persons and I decorate them as they are assigned to  new responsibilities. In that way one can use the same object to register all career assignments: 20-hour, 40-hour, dean who still teaches, and even represent some administrative employee who teaches in a specific course - without loosing his/her attributions in the university's management. Teaching, researching, and administering would be decorators that could be associated and dissociated to objects as the business processes require (pay-as-you-go).

Another point is that each decorator must keep a set of rules of association, which is responsible for allowing or prohibiting objects to be assigned to responsibilities. Each "decorated" object is also responsible for keeping a record of its responsibilities (bi-directional relationship). If a given object respects the rules of association of a given decorator, it can be decorated by it, allowing a very flexible way of creating new "types" (mix-and-match).

This reasoning is also valid for other concepts, for instance, a given manufacturing cell (Machine), can run new operations as new tools are attached to it. The same is valid for organizations, as new machines and persons are associated to it.

An interesting point on using decorators is when we think of business entities ruled by laws and regulations, such as contracts, government agencies, or even governmental economic measures: as regulation or environment changes, things can be added and/or deleted from the entities' scope. Rule/Law decorators can be programmed to function during certain period of time, detaching themselves from the decorated objects when this period is over. As an example of this last case, let's think of a tax refunding mechanism that is valid only during recession periods and for specific types of sales.

Of course, a problem is that we have to deal with a potentially big set of decorators. However, given the mix-an-match possibilities of decoration, this number is smaller than the number of classes that would be created to map all possible combinations. Using an extreme example, if I have three concepts that can, each one, be assigned to four responsibilities, we would have 12 classes - and possibly by the use of multiple inheritance. By using decorators we would have 3 classes and 4 decorators, or 7 entities to manage (in 12 possible combinations, of course).

Said that, the next step is to discuss the rules of association and details for implementing decorators.

Thursday, March 3, 2011

Enterprise Information Systems Patterns - Part IV

Changing the abstraction level
As expected, the high abstraction level brought by the use of only 4 concepts to try to define a whole system would bring side effects*. The basic one is that, if on one hand abstract concepts are highly reusable, on the other, they need to be extended a lot in order to become useful for representing concrete concepts, even the basic ones, such as Person or Product. I mean, a Node can be the abstract concept behind a Person object as well as a Factory object. However, you will need to provide a lot of extra code to make these concepts work. Given that I don't want to create lots of subclasses, the alternative is to use Decorators. However, in the same way, it is not good to abuse of a given technique, even if this technique is a well known design pattern. What I was going to do was create a lot of decorators instead of creating a lot of subclasses. In other words, I would be going against my main goal, which is to create a framework extensible by configuration, instead of programming.

Therefore, I stopped a bit to rethink the framework's abstraction level. One thing I realized is that it would be very hard to configure too abstract concepts, thus, by creating one more class level, I could find less generic but easier to reuse concepts. A side effect is that I can get rid of one more of the original concepts - Path. In other words, I shortened the width and broadened the length of the class hierarchy. The idea now is to use three classes: Resource, representing the production resources; Node, representing the entities that transform and transport these resources; and Movements, which represent transformations and transportations of resources. Each of these abstract concepts has three subclasses, representing two "opposite" concepts and an aggregator of these first two. Thus, the structure will be like this:

1)Resource()
-Material(Resource): product, component, tool, document, raw material...
-Operation(Resource): human operation and machine operation, as well as their derivatives.
-Kit(Resource): a collective of material and/or immaterial resources. Ex.: bundled services and components for manufacturing.
Comments:
a)Alternative terms such as Object (material) and Action (immaterial) can also be considered, however, the term Object can bring a lot of trouble in programming.
b)Another point is information, which is not material, neither it is something like operation. Document is used as the physical representation of information. c)Structural and Behavioral would be alternatives, however, they seem to be too academic to be used.

2)Node()
-Person(Node): employee, supplier's contact person, free lancer...
-Machine(Node): hardware, software, drill machine...
-Organization(Node): a collective of machines and/or persons. Ex.: manufacturing cell, department, company, government.

3)Movement()
-Transformation(Movement): an "internal" movement. Ex: transforming raw material, writing a report
-Transportation(Movement): a movement of resources between two nodes. Ex: moving a component from one workstation to another, moving a document from one department to another
-Process(Movement): a collective of transformations and/or transportations, in other words, a business process.
Comments
a) There maybe some confusion between Transformations and Transportations and Operations, but they represent different things. Movements use operations to transform or to transport resources. For instance, an industrial transformation uses a given machine operation during a given period of time to transform a given quantity of raw material into a component.

The next step is to define how to extend the basic concepts to make them work properly in concrete business processes representations, which will be discussed in the next post of this series.

* As I said in my first post of this series, if you need a "real-world" and flexible Python framework, give ERP5 - from which the basic ideas for this framework were taken - a try. Remember that this framework is a didactic one, therefore, some assumptions are simplified.

(The Change Log maps this series of posts to commits in GitHub)

Friday, February 25, 2011

The obvious, but yet ill applied, relation of Open Source and Auditability

In this post I comment two blog entries related to the use of open source software for auditing purposes in the banking area.


In the first one, Open Source Banking for Financial Stability, Jean Paul Smets  affirms that some serious problems presented by financial products not compliant to the regulations are caused by software systems which "are not subject to any kind of regulatory control and, therefore, may be used to circumvent regulations either through functional inconsistencies or through temporal inconsistencies." Therefore, he defends the idea of using a reference open source software, without getting rid of the software in which banks invested for decades. This reference, besides offering an auditable software representation of the regulations rules, would be used "to provide to regulation authorities a very accurate picture of [banks] activity, much more precise than what accounting can provide." - by importing the data handled by existing bank's software.


In the second post, Python Could Become the Language of Finance, Jonathan Allen informs that SEC is proposing that Asset Backed Securities should include a “program that gives effect to the flow of funds, or “waterfall,” provisions of the transaction.” Waterfall in this case "refers to how some bonds are broken into levels, where those who bought into the higher level must be paid off before the lower level sees any money." The interesting part is that SEC is planning to mandate Python as the language for building this reporting tool, because it is open source, uses a standalone interpreter, and is supported by many platforms, including proprietary software such as .NET.

Although the first post is an opinion, the second isn't, and both present good arguments to motivate regulatory organs to use more open source for auditing tasks in the financial area, and, why not say, in other areas. The use of open source reference softwares would (i) help people compare their implementations of regulatory rules with a standard implementation, (ii) provide cheaper software that could be customized for specific adopters, while following the regulation, (iii) provide automated tools for regulatory agencies to audit data. In that way, maybe the number of financial scandals could be reduced, as well as regulatory work in other areas, such as environment, could be simplified and streamlined.

Sunday, February 13, 2011

Enterprise Information Systems Patterns - Part III

After my last post, the code in Github changed many times, since I was experimenting with both the basic concepts and the testing stack we are using. We decided not to write features (BDD's Given-When-Then constructs) for the configuration of Resources, Nodes, and Movements, since it would be overkill (as you can see in the sketch branch, from December 12, 2010): by reading the specs anyone can understand and check the functioning of configuration.

The comments below are based on the last implementation (January 09, 2011), which doesn't implement Path nor the parsers for the configurators. We are considering writing features for Path configurations, because Paths represent business processes, and therefore, it make sense to have high level descriptions of requirements.

Currently, the proposal is based on two base classes, from which all others derive. The first one, Configurable (previously known as Maskable), is the base for building the abstract concepts, and the second one, Configurator, is used for providing a "friend" class for each concept which is responsible for managing the configuration of the concepts' instances.

Besides the basic attributes, Configurable abstract class holds a "To Do" method, which is simply a placeholder to remember that a method for doing "basic things" with instances should be implemented by subclasses. Configurators on the other hand must be able to parse, persist, and retrieve configurations to be applied to concept's instances. They are multitons, in other words, for a given key, composed by configuration's mask and version attributes, only one configurator object may exist. Therefore, every concept class, such as Movement, has a configurator class, such as Movement_Configurator.

Configurations supply the following attributes for instances (spec files provide complete configuration examples):
-Mask:  defines a specific type for the instance. Example: payment (a payment movement).
-Version: holds the version of the mask, multiple versions can be in use at the same moment. Example: monthly (monthly payment).
-Category: associates a category to a specific concept. Example: financial (monthly payment is a movement of category financial).
-Description: provides a description for the key mask+version. Example: ' a monthly payment from a organization to a person'. Note that, in this example, organization and person can also be categories.

Right now the implementation is in a very simple state, the next step is to implement Path, which is expected to reveal more requirements for making this framework more usable. For testing, we are going to use some real-world workflow implemented in ERP5, applying the mappings presented in the technical report "Mapping Business Process Modeling constructs to Behavior Driven Development Ubiquitous Language" to describe the workflow using Given-When-Then.