It’s Not Your Fault – Blame the Designer

Posted on by
2

Most people don’t realize how often they actually do mistakes. Every wrong move or thoughtless action, even if you can recover it immediately, is still a mistake that could have been possible to perform flawlessly. In general, people don’t admit doing errors, especially publicly in fear of becoming shamed. Six Sigma suggests as the source of the most of the cost in the products the so called Hidden Factories, or error corrections that are performed by the employees without informing the management, to avoid taking the blame, but incurring nevertheless the cost of the error. However, the belief that the mistake that you just made would be your fault is actually not true! If you just made an error, it’s actually not your fault! Instead, you should point your finger to the designer.

At Toyota the Kaizen continuous improvement and learning organization uses the following framework for blaming and learning from mistakes. You can adapt this philosophy also on your personal life, but I discuss it from the entrepreneur’s point of view.

1. Blame the Instructions

If you don’t have instructions how to perform a specific task, the work design and management is to be blamed. The definition of the management and entrepreneurs is actually not do any work, but to manage the organization of work, meaning how exactly you should perform your tasks and which tasks should be performed in the first place. The work of an entrepreneur is to organize the factors of production so that you can convert ideas into practice filling out a market niche. The work of the managers is to find optimal organization of the work for a given problem. The worker should not be interested of the management of the work, since it’s not his primary interest, as long as the company stays afloat and employment is guaranteed. Similarly, a manager or entrepreneur who is not interested in the continuous improvement of the work processes is not performing his job properly.

The current and western management nomenclature tends to omit the first and primary factor of production as irrelevant – the standardization of work. However, the as any basic economics book can tell you straight-on, without standardization of work there is no economies of scale and the work remains ad hoc in nature and economically unfeasible. In the western countries the emphasis is put on the innovation, or the Deming’s PDCA -cycle, how to improve the current work process by radical innovation. Unfortunately most often the basics are forgotten in the fuzz of innovation and the management fails to standardize the work after or before the innovation, resulting in utter chaos and high costs. No wonder that the jobs are continuously being outsourced to China and India.

Secondly, if by a miracle a standardized work process would exist in your organization, the instructions might well be incomplete, inaccurate and ignorant of the most common error modes. The first corrective action at Toyota is aimed thus on improving the instructions so that the by following them the work can be performed succesfully.

Yet a more advanced philosophy for standardization is the Six Sigma -approach that tries to identify and mitigate the sole possibility of producing an error. The Japanese call this principle as Poka-Yoke, or error-proofing. The idea is that by eliminating the source cause for errors the work process can be performed flawlessly producing significant quality and cost improvements.

2. Blame the Machines

If despite improving the instructions you still make them, you must turn the look on the machinery and tools that you are using to perform the task. For example if your computer is crashing and preventing you from performing your work, it’s not the fault of the instructions or yourself, but of the computer. Thus you should find a new tool or machine design that makes it impossible to perform such an error.

The Designed for Six Sigma (DFSS) -method tries to achieve exactly this. 80% of the quality problems can be tracked back to the design board, and thus it is not possible to reach the higher levels of quality (5 and 6 sigma) without taking the high quality in account already on the drawing board. The DFSS tries to eliminate the potential for producing Critical-to-Quality (CTQ) error in advance by for example preferring usage of proven components, analysing the potential error modes (FMEA) and the Voice of Customer.

The Japanese have developed a 14 level model of intelligent machine automation (Jidoka). The idea is to convert the machines to automatically detect errors in themselves. The idea comes from Toyota’s founder’s Sakichi Toyoda’s automatic weaving machine that automatically detected a broken loom. Six Sigma suggests of using simple automatic pass/fail -gates for investigating the quality level on a production line. The same idea is called as Test-Driven Development in software engineering, where you before starting of the coding define the end product by writing automated unit and acceptance tests.

3. Blame Yourself

If you are not able to follow the instructions or the standardized process, it’s your fault. In this case Toyota considers a task for you that you are able to perform according to the instructions. However, before blaming on the human factors, one should consider the two previous steps if they have rendered the work inhumane to perform, and if the blame can be still be placed on the Work and Machine Design first.

Of course, if you are unable to follow the Kaizen continuous improvement -cycle, the blame is truly only on yourself :)

Start-ups! Beware the Corporate Managers

Posted on by
2

Beware hiring of the Corporate Managers, especially the 50y+ ones that you most commonly tend to contact you spontaneously, in the Start-up -context. Typically they are suggested to be hired on your board, as the VCs typically want to strengthen the team with some seasoned expertise. Geoffrey Moore describes one (and only one) use for corporate managers – if they are from the target customer company or industry, they can act as successful sales managers for acquiring the first successful sales. However, they bring with some unexpected and harmful backage that the entrepreneurs should be aware of. I would not recommend to use the corporate managers for any other task than for the one described by Moore. Here are some reasons why:

Failure to Fail

The problem is that while the start-ups should be capable of failing fast, the middle-aged corporate managers often have worked their whole life in a culture where failing is not tolerated, especially in the upper-management. Thus, they will to the end and also afterwards disagree from having made any bad decisions, or changing their opinions even though the objective data would show the opposite. In the large corporations the vast resources enable the managers to dismiss their mistakes silently, but for the start-ups the consequences can be dire indeed as their resources are thin. The habit gets the worse the older the manager is. Finally they trust only into their “instincts” and dismiss all other information. In my personal example the hired-in corporate management insisted of using the just acquired funding for a large advertisement campaign on the launch, in direct contradiction to the original marketing plan, which was used to rise the funding in the first place. Due to the hastening of the launch, it happened in the middle of the holiday season and in the wrong media, with no effect. Too bad that the investment was spent out in a record time. The board had even more bolder and aggressive ideas of wasting  the money even faster and to even more inefficient targets, but they couldn’t be implemented due to the cash crisis.

Lack of Functional Skill

The second problem with the corporate managers is that they have customized in having a large support organization (payroll, accounting, marketing, sales, R&D) to which they can delegate their tasks to, and assume that these functions hold sufficient expertise to perform their tasks adequately. However, the start-ups are different. Most often the only expertise you have is what the entrepreneur has between his ears. By introducing the corporate manager, the costs can quickly sky-rocket beyond any reasonable level sustainable for start-ups, especially when the corporate managers have to use outsourced resources. The problem lies within the assumption of the corporate managers that they can use the outsourced accountants as they have used to utilize their in-house counterparts, delegating every smallest detail without ever gaining full understanding what the task actually consists of. The outsourced accountants are extremely happy, as they can start to invoice the start-up at high rates (60-150 EUR/USD/h) for performing routine work that should be paid at 10-20EUR/USD/h).The large support volume of the large corporations enable them to provide in-house support functions with lower cost than by hiring them from outside. Too bad that this is way too expensive for the start-ups.

Disregard of Details

One common habit for all corporate managers is their lack of interest in detail. This is actually the essence of management – a manager should not be an expert in the given field, but to hire the right experts. The large corporations and public organizations practice management daily – and hire experts and consultants for high fees to solve their specific problems. Also the education is nowadays tuned up to fulfill the needs very specialized skill needs of the corporations. If you are an accountant, it makes no sense to know anything about IT-systems or marketing.

This is, however, not the case for small companies. An entrepreneur who disregards the small print in the term sheet, or trusts an inexperienced or a wrong lawyer, even when paying sizable fees, will in the end pay dearly for his ignorence. Timmons, Spinnelli and Zacharikas calls this phenomenon in the magnificent book “How to Raise Capital” as the Legal Circumference. An ignorant corporate manager will sign deals that are not only very difficult to live with, but also create tight limitations and constraints on future choices, which is potentially disastrous.  Entrepreneurs simply cannot rely on attorneys and advisors to protect them in this vital matter, but have to put their skins in the game. A corporate manager on the other hand will thrive to avoid all responsibility.

Taking for Granted

The next shortcoming is closely related to the previous one. By having lived in a large corporate organizations, someone else has built the structure of the company to function long before the manager joined the organization. The start-ups are a blank sheet – on the first day of operations you don’t have payroll, accounting, bank accounts or even idea exactly how each of these necessary processes should be organized. Neither do the corporate managers, they have always lived in a world, wherein all the processes have existed and been functional. They have never given a though to the idea, how they they are actually organized, as the processes have worked well enough for so long time that nobody remembers anymore, how they were setup initially. Too bad for an entrepreneur, who wants to delegate this task. You simply can’t. You will have a hard time finding anyone, who would have even a slightest idea, how the basic processes should be bootstrapped from the scratch. Especially do not ask help from the corporate managers or outsourced accountants/lawyers/other specialists, as they are not experienced in these questions.

Inability to Adapt

Due to the lack of true understanding and lack of expertise of domain skills, the cost of hiring a corporate manager will be high – since he is an expert in management. Thus he will utilize his only skill, and delegate everything regardless of the cost. What the start-ups really need instead is experts, who have the domain skill (accounting, legal, coding) and can contribute his skill in exchange to stock options or any other non-monetary ways than by paying in cash. Hiring a manager to manage the outflow of the cash is suitable only for large corporations that can bare the overhead. The corporate managers are truly unable to operate in an environment, where every cent matters. Thus, the entrepreneur should never let his eye from the efficiencies of every detail of the operation, unless it’s so profitable that you can sustain the overhead-waste of large corporations (revenue less than 2M EUR/USD). Although the managers might benignly be offering their help to the start-up even for free, the managers are unwilling to learn any skills that would make them actually useful. Instead of wasting your time in talking with the managers, who can’t perform any valuable work, look for skilled specialists, buy some books and learn the basics of litigation/tax law/marketing by yourself. Because nobody else will do it for you, and you don’t have the money to hire a specialist for every task, unlike the large corporations.

Lack of Commitment

The corporate managers have accustomized themselves also to the habit of not having responsibility of anything. Rather than being risk taking (as an entrepreneur by definition has to be), the managers are risk-awerse, trying to identify personal risks and continuously finding ways to distance personally from potential consequences. From the entrepreneur’s point of view the corporate managers are like rats, who eject themselves overboard immediately if some danger can be perceived. For example in CMAX, the VCs demanded of hiring a new board member. However, the chosen board member after first accepting the task and having paid for the task, communicated after receiving the up-front (government allowance grant) payment that he is not really interested in having any formal position in the company, but would rather be a non-voting advisory board member.

The chairman of the board (also a 50y+ corporate manager) performed a coup by agreeing with the minority shareholder VCs to dismiss the entrepreneur and the CEO (which was accepted by the entrepreneur in belief that the work load would easen) and starting to manage the company (by the corporate management-style as described above). However, as the lack of functional expertise, the true cost of delegation and the inability to adapt became evident, the entrepreneur had no choice, but to force the dismissal of the previous board and taking control of the company once again. The lack of commitment is not something that a start-up needs. Ultimately the captain is required to steer the ship around the rocks. I suggest that you arm the mouse-traps well in advance and get rid of the corporate rats before they eat up your grain-stock permanently.

Never Let a Corporate Manager to Pitch for Finance

The one more disastrous failure came after the coup, when the corporate manager participated the funding pitch. By being ignorant of also this functional expertise area, he slipped to use the potential funding for advertisement and sales. Unfortunately the funder was a government agency that was prohibited from granting funding for any kind of marketing activities, and this critical funding application was denied. The entrepreneur, who had successfully raised 8 previous funding rounds had to sit on the side in disbelief of the performance of such a stupidity, trying to unsuccessfully to save the loss. The teaching of this story is that due to the lack of commitment, lack of any kind of domain expertise except for the management/delegation skill, the managers are simply unable to perform the story telling required for a successful pitch. The financiers are very good at discovering the applicant has actually no idea what he is pitching for, yielding only quick disapprovals of applications.

The habits of the corporate management are very difficult to change. The entrepreneurs and the corporate managers are two different breeds that live on the different sides of the Chasm (you know, the Geoffrey Moore’s one). The entrepreneurs are the visionaries and the technology enthusiastics, while the managers are the pragmatists and conservatives, not believing anything that is not referred to them by other managers. Thus the managers are viewed are actually a stubborn flock of self-referential sheep, who are very disastrous to be involved in any start-up -company. Instead of hiring corporate managers, you should staff your board with other seasoned entrepreneurs, who have actually been crossing the chasm before, and know the travel on the both planes. Be warned.

Is the Current Financial Crisis Result of a Disruptive Innovation?

Posted on by
Reply

As everybody knows the financial credit crisis has struck the World in 2008-2009. Trillions of dollars have been spent in supporting the in-trouble commercial banks and other large corporations. Companies report billions in losses and hundreds of thousands are laid-off. The generally accepted reason for the current crisis is the US sub-prime mortgages that nobody seems to have been able to predict they will default. Despite the fact that the banks are traditionally considered to be very risk-awerse. Makes me to wonder!

Harvard Business School professor Clayton M. Christensen gives another explanation, what is really happening. Actually the bank crisis is not about the crisis in the economy, but an introduction of a new disruptive technology called credit scoring schemes that have struck hard into soft belly of the traditional loaning businesses of the incumbent commercial banks. The credit scoring schemes are algorithms and formulas that let you to evaluate credit-worthiness of a loan applicant. The new players who are using this new technology are not banks, but new specialist lenders such as GE Capital and the others, that push the traditional banks upwards towards the higher margin investment banking. However, the large banks are starting to feel the air getting thin in the higher market segments, when the new specialist players are eating up their traditional core-businesses. It started from the consumer loans, moved into the automobile loans, credit cards and starting to reach the small business loans.

How do the banks react when their traditional businesses are evaporating? To cope with the decrease in margins they have to push the costs down by marging into larger units. This has happened already for decades. The Bank of America mergers with Merrill-Lynch, Citigroup acquires Morgan-Stanley etc. According to Christensen this is an ex-post reaction by the high-end players to a market disruption that has already happened. The new players have pushed the commercial banks to investment banking and to merge, since there is not anymore enough volume to sustain them all.

The idea of Christensen’s Disruptive Innovation -model is that the new disruptive technologies are introduced to the market as inferior alternatives and thus are ignored by the incumbents. However, after the disruptive technology evolves, it suddenly reaches the good-enough requirements of the customers, and overtakes the markets with higher efficiency business models. The incumbents try to protect their eroding high margins, but in the end are unable to compete with their high-cost business models with the new rivals.

As we remember, the credit crisis was launched by the US sub-prime mortgages. It seems that the traditional banks are truely in trouble with this disruptive technology, as the root cause of the crisis is the inability of the banks to evaluate the credit worthiness of the re-packaged sub-prime loans. The losses of the high-tier commercial banks accumulate in trillions of US dollars! On the same time the new market players are cashing in steady profits.

Bad for the banks, but good for GE Capital. But how does it affect the consumer? The bad news is that due to the trillions of dollars of Government bail-out money, the tax payer has to actually pay for it all. And the even worser news to the tax payer is that he is paying for nothing – according to Christensen’s model the large incumbent banks will be anyway be forced to merger and shrink to the oblivion, until there are no old banks remaining. The cost will be enormous, especially, if the inevitable technological revolution is stalled by taxing the consumers. The bail-outs are only pro-longing the inescapable with an ever increasing cost for all the parties involved (=everybody), when the dying incumbents are artificially resprirated.

It gets me to think, who are really influencing the political decision makers? Who has the largest amount of money in the world? The bankers of course, for the time being. Inevidably they will, however, shrink to second class players, when the new cadre of financial institutions rise. The downside is that the bankers are now puffed up with such amounts of capital that it will take decades before their final collapse.

Read More:

Financial Disruptions http://www.innosight.com/innovation_resources/insight.html?id=487
Credit Scoring http://www.bba.org.uk/bba/jsp/polopoly.jsp?d=135&a=6612

By Antti J. Hätinen / http://pharazon.blog.com 2009-08-05

The Inmates are running the Agile Asylum

Posted on by
Reply

Here is a great interview of Alan Cooper, the designer of Visual Basic & Visual Studio and the author of the Goal-Driven Design (GDD), a close relative to the GUIDe by Sari A. Laakso:

http://www.infoq.com/interviews/Interaction-Design-Alan-Cooper

Agile is not about productivity, it’s about the core of motivation of the developers. The traditional management of the software projects usually lack understanding what is going on, set unrealistic objectives, drive for low quality and make the work of the developers miserable. While the industrial-era management doesn’t really understand what’s going on, the developers have filled out the vacuum by managing themselves. Instead, the knowledge workers are not motivated by money or following the schedules, but doing good (or great) work. Thus the rise of the Open Source, over there people can do as good products as they wish :) If the industrial management techniques makes this impossible by managing the knowledge workers as industrial workers, then the developers are not pleased either. Because nobody is really managing the development work, the Agile has risen from the ranks of the developers to fix the management problem.

Alan Cooper claims that the process of the interaction design is quite similar to the agile ways. The key process is to reflect on the business problem before day zero of the start of the development. The key thing is to have deep and profound understanding of the business process.

Tasks are not Goals

The important thing is not what tasks the users do, but what is the end state. The design process is to redesign the tasks so that the goals can be achieved easier. By designing based on the tasks the result will be a Dancing Bear :) . It dances, but not very prettily. The objective of the interaction design -school / GDD / GUIDe is to make the bear to dance well!

In software there is no economics of scale

In the old times the main driver of the business was to get the unit costs down. However, in the software industry the maintenance costs are zero or very low, but the development cost is rather high. The economics of the software are prfoundly different to the economics of the manufacturing. Driving the cost down just drives down the desirability of the design. The most important thing is to worry how you can elevate your number one goal. The business men should think only about how to increase the business value and quality instead of reducing the cost.

Iterationless Scrum III

Posted on by
Reply

I found a second article describing an iterationless agile SW process. This time it’s Dan Rawsthorne from Danube introducing the new Scrum III. He says that the Sprints should not be anymore strictly obeyed, but the user stories can enter the WIP (Work-in-Progress) -queue accriss the Sprint boundaries!

It seems that he wants to move Scrum more to the direction of the Lean by replacing the Sprints by an iterationless Kanban/Just-in-Time pull-system. I found already one article earlier about the same idea. For me this seems rational, since the original Lean really actually has no iterations, they are just remnants of the legacy SW Engineering idea of Iterative and Incremental processes. A true Lean process is always one that releases one feature at a time, with a constant pace, regularly, and carries no Technical Dept or half-finished features. The optimum way is to use one-piece-flow meaning working only on one job at a time. Multi-tasking has been scientifically been proven to be sub-optimal since you introduce waste by increasing the unnecessary waiting time, making the cycle times slower and thus increasing the inventory / WIP. If you don’t release immediately when your feature is completed, you essentially store your code in a non-productive warehouse. You have paid the salaries, but haven’t earned a cent out of the investment. The situation gets even worse if you have happened to code some bugs in – the waiting time before release increases the cycle time for finding and fixing the bugs. The faster the feedback cycle time, the better, thus you should release in small batches and immediately when you have completed some code worth of business value. In the Extreme Programming the 2-week fixed iterations are suboptimal in this sense, although you have the Release Often -rule. However, working on two or more user stories simultaneously increases the real cycle time and thus cost.

I’m not currently managing any SW Engineering team, but an iterationless XP/Scrum is definately something I will use on my next project. The only thing to do is to work only on one user story at a time. Shouldn’t be too hard to get it to work :) .

Javascript Memory Leaks

Posted on by
Reply

Back to the code :) I got assigned to debug a report that the Microsoft Virtual Earth crashes IE6. Unfortunately it took me quite a while to get the environment up to be able properly debug the problem (= I had to install one Windows machine from the scratch). After Googling a while and testing some other tools, I found the IE Memory Leak Detector tool from the MSDN, that looked very promising.

My first step was actually to build a Selenium test case that navigates the map (left-down-right-up in a loop), zooms in and zooms out. Running this test in a loop for 100 times revealed a some differences in memory usage between the browsers. The IE6 started from roughly 58MB, but after a hundred iterations ended up to use 138MB of memory. Opera started to leak actually even faster than IE6, but used only roughly 125MB at the end of the test loop. Finally I tested the FF3.5, that used 88-157MB of memory during the loop, but seemed to garbage collect properly occasionally and it seemed not to leak noticeably.

Finally I got the new Windows test machine equiped with all the drivers, upgraded, activated and installed with Selenium RC, Java and the rest of the tools. Based on the test runs I suspected that the leak must be in our own javascript -code, while googling revealed a number of articles describing how you should set the parameters null after using and the usage of closures. However, the IE Memory Leak Detector gave this kind of reference to code that caused the leak:

function _f1248273915185(){
return {“__type”:”Microsoft.VirtualEarth.Engines.Core.ImageryMetadata.PublicTypes.BirdsEyeSearchResponse”,”Scene”:{“S”:11653832,”O”:0,”Q”:”12012021101″,”RI”:2504,”L”:20,”Fcx”:16,”Fcy”:12,”Hcx”:8,”Hcy”:6,”QA”:0.20479389053054289,”QB”:-11.339952179270028,”QC”:-66060.4659001751,”QD”:0.51723632379069606,”QE”:-27.329727071417665,”QF”:-159324.1954204863,”QG”:0.0086047506233842379,”QH”:-0.45437445343971539,”QI”:-2647.8944163784549,”XA”:-99.125040347204759,”XB”:-40.257039373405007,”XC”:4895.2732690969769,”XD”:-5.057457829331435,”XE”:97.444190848983354,”XF”:-5737.0562779899165,”XG”:0.00054572923124807408,”XH”:-0.016852089141664729,”XI”:1},”ResponseSummary”:{“Copyright”:”Copyright 2009 Microsoft and its suppliers. All rights reserved. This API cannot be accessed and the content and any results may not be used, reproduced or transmitted in any manner without express written permission from Microsoft Corporation.”,”StatusCode”:0,”AuthResultCode”:0,”ErrorMessage”:null,”TraceId”:”5b584e34-1354-4571-85f6-daf69d27f64e|AQ23817″}};}
if(typeof closeDependency !== ‘undefined’){closeDependency(’1248273915185′); }

After a bit more googling, I found that this is a response object of a fetching function that seems not to unload properly.  While I was also unable to find a reference to this code (our code seemed to use the SOAP-interface for map loading), this got me to think that the bug could be on the Microsoft’s side of the code. I need to talk to our Javascript gurus :) .

Network Centric Software Engineering

Posted on by
Reply

I was reading the Cooper Journal of Interaction Design when I stumbled on a few articles on Sensemaking and Network Centric Operations (NCO), a new military organization doctrine that tries to use information technology to facilitate operations of geographically dispersed units and to enhance operational effectiveness compared to a traditional force. For me this sounds relevant to software engineering, too, so I got interested. Wikipedia noted that a similar approaches have already been adopted by the UK and Swedish military.

The idea of the NCO is to enhance the situational awareness by improving the information exchange between the troops. This includes perceiving the environment, comprehending its meaning, and projecting the status in the future.  As a software engineer this sounds very familiar. The SW projects rarely have situational awareness even when all the developers are present on the same floor or even when they are sitting side by side. When the project is introduced with the off-site customer or the outsourced Indian developers, the information exchange problems tends to escalate even further. Why it is so difficult for the software engineering teams to exchange information even between the developers? One article on the Cooper Journal discussed the similarities between the SW Engineering and Movie production, but one commentor noted that the most significant difference is that in the movie casting the director has the final say of everything, while in the software projects nobody has the technical and visionary ability to comprehend all parts of the system or communicate on all topics.

The NCO approaches the problem differently. The key process of sensemaking involves all levels and perspectives of the organization and is grounded on a robustly networked force that is capable of information sharing. The second tenet assumes that the information sharing and collaboration enhances the quality of information and the shared situational awareness. This leads into self-synchronization of the units, that leads finally into a dramatical increase of mission effectiveness.

The sensemaking process by the U.S. Department of Defence starts by

  • Forming an awareness of key elements relevant to the situation. This entails knowing “the who, what, when and where.”
  • Forming an understanding of what it all means in some bounded context, based upon past experiences, training, education and cognitive capabilities. This entails:
    • Forming hypotheses and making inferences, i.e. generalizations (predictions or anticipations) about future events.
    • Forming a sense of the implications for different courses of action.
  • Making decisions by:
    • Generating alternative response actions to control the situation.
    • Identifying the objectives, constraints, and factors that influence the feasibility and desirability of each alternative.
    • Conducting an assessment of these alternatives.

The enhanced collaboration and higher quality of information enables people to make better decisions. I can see directly, how this can be applied on a distributed software engineering project. The practices such as a Daily Scrum already implements partly the same purpose, but it should be taken further. I envision that a NCO software engineering team would utilize tools like Twitter to continuously update the status of what they are doing, what are their problems, what things should be known about and estimates about the remaining effort. Yesterday I watched a Apollo 11 document where the lead flight director Gene Kranz explained how he simultaneously listened to 7 simultaneous communication loops, but with training was able to focus on the one communication that was important for performing the current task. Similarly the twits could be a way to exchange information and meaning in a distributed SW project. However, a key enabler is to identify the communication language, or the common concepts everybody should understand. Another point is to define the key elements of the situation. The NCO -model seems to promote also the learning organization -dogma by giving permission and tools for the team members to understand and challenge the  knowledge of others (or the situational awareness). Often the true ground-reason for project failures is not the tools or the people, but the inability to acknowledge the current situation and react to the changes.

The IT industry is surrounded by all kinds of communication software, but still unable to utilize or leverage it for meaningful communication. Perhaps the problem is with the cognitive models of software team organization and new models such as the NCO is desperately needed.

Improve your Architecture skills with PHP_Depend/jDepend

Posted on by
Reply

Installing PHP_Depend

I found an article about creating packages and to improve the modularity of your code. I’ve been always wondering the meaning for packages, but haven’t found any good explanation before stumbling on jDepend (PHP_Depend). This seems to be great stuff for honing your object architecture design skills!

What the jDepend/PHP_Depend tries to say is, instead of putting all classes into one package, or making a separate package for exceptions etc, you should try to think packages as modules. JDepend and PHP_Depend helps you to check how modular your code really is. Read more here.

pyramid

INSTALLATION

Login as root and type

pear channel-discover pear.pdepend.org
pear install pdepend/PHP_Depend-beta

Then go to your source code directory and run

pdepend –summary-xml=/tmp/summary.xml  –jdepend-chart=/tmp/jdepend.svg –overview-pyramid=/tmp/pyramid.svg .

Then go to /tmp with your web-browser and you should see two graphs. The pyramid shows suggestions and ratios for example of the NOP (Number of Packages) and NOC (Nubmer of Classes). According to one scientific article you should have an avarage of 17 classes in a package. If you have less than 6 or more than 27, then you should perhaps consider refactoring your code. See all the explanations from here.

The second graphs shows the (perpendicular) distance of packages from the idealized line A + I = 1. This metric is an indicator of the package’s balance between abstractness (A) and instability (I). A package squarely on the main sequence is optimally balanced with respect to its abstractness and stability. Ideal packages are either completely abstract and stable (x=0, y=1) or completely concrete and instable (x=1, y=0). The range for this metric is 0 to 1, with D=0 indicating a package that is coincident with the main sequence and D=1 indicating a package that is as far from the main sequence as possible.

Here are the explanations:

  • Abstractness     The ratio of the number of abstract classes (and interfaces) in the analyzed package to the total number of classes in the analyzed package. The range for this metric is 0 to 1, with A=0 indicating a completely concrete package and A=1 indicating a completely abstract package.
  • Instability     The ratio of efferent coupling (Ce) to total coupling (Ce / (Ce + Ca)). This metric is an indicator of the package’s resilience to change. The range for this metric is 0 to 1, with I=0 indicating a completely stable package and I=1 indicating a completely instable package.
  • Efferent Couplings     The number of other packages that the classes in the package depend upon is an indicator of the package’s independence.
  • Afferent Couplings     The number of other packages that depend upon classes within the package is an indicator of the package’s responsibility.

PHPUnit 3.4, –process-isolation and phpUnderControl

Posted on by
7

Recently on one project I tried to execute the Zend Unit Test Suite, but stumbled on a memory exhaustion problem. If I tried to set the PHP memory limit above 2GB, my computer crashed, probably when it tried to swap. I realized that if the 2GB memory on my desktop machine is insufficient, the 512MB on the cruisecontrol machine will never be enough for running the whole test suite.

On the CMAX.gg -project our coders implemented a custom shell-script based cruisecontrol system, that ran each SimpleTest test case through the web browser. On the current project I decided to use open source tools such as phpUnderControl and PHPUnit instead, so that I could use a readily available tool instead of trying to parse a new tool from the scratch again. Also, it seemed that the PHPUnit is more active and has nowadays more support than the SimpleTest.

After configuring phpUnderControl for several days, I realized that our old system was actually better than what is currently available. For example you need to restart Cruisecontrol.org Tomcat webapp each time you make changes to your config files, I needed to create separate scripts to check and restart Cruisecontrol when it crashes about once per week, and it uses a huge amount of memory and disk-space. I managed to reduce the disk-usage by disabling the phpDoc -target on each build. Yet more disappointments came when I realized it is not possible to run a single test case through the Cruisecontrol/phpUnderControl -interface and the graphs and the reports were almost unusable. If you are aware of the Agile practices, you really want to have only an overview, where you see the number of failed tests and the total number of tests. The phpUnderControl doesn’t sort the tests failed first unlike our own system does. I also realized already 5 years ago, that it is quite useless trying to draw a graph where you have the ratio or percentage of failed tests, since quite quickly you end up drawing a line with 99.9% and 99.8% pass ratios. A better graph is the absolute number of failed tests.

After Googling a bit, I ended also up to the PHPUnit Changelog, that advertised a new –process-isolation switch. The drawback was that I needed to upgrade from PHPUnit 3.3 to 3.4. After some experimenting I managed to run the tests, but they all failed to an unspecified error at an unspecified location like this:

124) Test_Ext_Zend_Zend::testSetState
RuntimeException: Warning: require_once(trunk/ext/Zend/tests/Zend/AllTests.php): failed to open stream: No such file or directory in trunk/test/Test/Ext/Zend/Zend.php on line 5
Call Stack:
0.0090    1206200   1. {main}() /home/pharazon/workspace/trunk/-:0
0.0093    1223456   2. require_once(‘/home/pharazon/workspace/trunk/test/Test/Ext/Zend/Zend.php’) /home/pharazon/workspace/trunk/-:5

As you see, the stack trace says that the error happened at an file called “-”, a quite hard trace to debug. After about a day of debugging I managed to get error messages to show by printing them out from the PHPUnit/Framework/TestCase.php ( print_r($job) ). After this I managed to figure out that PHPUnit creates dynamically a new PHP -file from a template for each TestCase. The basic Template is located at PHPUnit/Framework/Process/TestCaseMethod.tpl.dist . After more debugging, I managed to get most of the tests working by modifying the template and adding a bootstrap -file to setup the environment and the constants. PHPUnit –process-isolation uses reflection, var_export() and the magic __set_state() -method to serialize the TestCases on the fly. One hindrance was that the Zend -classes didn’t implement the __set_state() -method, so I was required to add them. The hardest ones to decipher were the Zend Translate -related classes, where just calling the constructor wasn’t sufficient. Finally I managed to create a function like this:

/**
* Magic reflection function to unserialize var_export’ed objects
* to be used with phpunit –process-isolation
* See more from http://www.thoughtlabs.com/2008/02/02/phps-mystical-__set_state-method/
* @param the same than for __construct()
* @author Pharazon
*/
public static function __set_state($data) {
return array_shift($data);
}

Compared to our own CMAX Cruisecontrol, it seems that the PHPUnit –process-isolation is a quite cumbersomely implemented and restricted. For example the current system doesn’t still enable running the tests on several cores, but only one thread is being used. Our Apache -based test runner was already 5 year ago able to run a multiple test cases per machine, and actually also distribute the running of tests across multiple test nodes. It’s quite easy through a http-interface, when Apache takes care of the multi-threading. Additionally the old system submitted the results on a test database. I think PHPUnit can do it also, but haven’t implemented it yet.

Next I started to package the solutions, but realized that using a custom statically modified PHPUnit and Zend was not probably the best idea in the world. Luckily I stumbled on the TestCase->prepareTemplate -function and realized it would be easy to create a subclass MyTestCase extends PHPUnit_Framework_TestCase that would implement the modifications without need for altering the PHPUnit source code statically. There are two changes required. The first problem is the loading of the bootstrap. The default template checks the loading of the file defined on the –bootstrap -command line parameter after importing of the globals. The problem was that the GLOBALS included the Zend -classes that were not yet introduced, since the bootstrap was not yet loaded. To solve this chicken-egg -problem the only solution was to create a custom TestCaseMethod.tpl that had a static bootstrap loader before introducing the GLOBALS. Additionally I added a new parameter called ‘bootstrap’. Here is the new MyTestCase:

<?php
/**
* My Test Case that extends PHPUnit 3.4 TestCase, featuring
* – custom preparation to support bootstrapping with –process-isolation
* @author Pharazon
*/
class MyTestCase extends PHPUnit_Framework_TestCase {

/**
* Performs custom preparations on the process isolation template.
*
* @param PHPUnit_Util_Template $template
* @since Method available since Release 3.4.0
* @author Pharazon
*/
protected function prepareTemplate(PHPUnit_Util_Template $template)
{
//use custom process-isolation template
$template->setFile(ROOT.’/test/MyCaseMethod.tpl’);
$template->setVar(array(‘bootstrap’ => ROOT.’/test/testBootstrap.php’));
}
}

Then I changed all tests to extend MyTestCase instead of the PHPUnit_Framework_TestCase. Here is also the modified MyCaseMethod.tpl template:

<?php
set_include_path(‘{include_path}’);

//require_once ‘{bootstrap}’;
require_once ‘{filename}’;

function __phpunit_run_isolated_test()
{
$result = new PHPUnit_Framework_TestResult;
$result->collectRawCodeCoverageInformation({collectCodeCoverageInformation});

$test = new {className}(‘{methodName}’, unserialize(‘{data}’), ‘{dataName}’);
$test->setDependencyInput(unserialize(‘{dependencyInput}’));
$test->setInIsolation(TRUE);
$test->run($result);

print serialize(
array(
‘testResult’    => $test->getResult(),
‘numAssertions’ => $test->getNumAssertions(),
‘result’        => $result
)
);
}
{globals}

__phpunit_run_isolated_test()
?>

I was happy when I got all the tests to pass :) . But only for a moment…. Next I tried to get the tests to run on the CruiseControl / phpUnderControl -instance, but noticed that the PHPUnit 3.4 doesn’t support anymore PMD or the CodeCoverage -tools, but they have been extracted to separate processes. Doh! But it’s yet a new story that you will get later :)

Lean & Agile Software Engineering

Posted on by
Reply

I got recently my second thesis work completed, during which I managed to read quite a lot management literature from the 80′s and even the magnificent Henry Ford’s “My life and work” from 1922 describing most of the Total Quality Management principles almost 90 years ago. Yet we are still not using the practices that produced one of the greatest industrial enterprises to the date.

Another fundamental and nicely written book is Eliyahu Goldratt’s The Goal describing the management process of Theory of Constraints. According to Goldratt, the mission of a company is to maximize the throughput (sales to the customers), while simultaneously minimizing the operational expenses and the inventory. Every system has a single bottleneck that constraints the throughput. By locating the bottleneck one can improve the throughput, but improving any other area will not improve the performance at all. This principle is the same as the Extreme Programming’s Leave Optimization till last. Otherwise you end up in tuning parts of your code that will not improve the performance at all, but make your code harder to understand.

After reading these and a few more books the agile software engineering principles suddenly appeared much clearer and theoretically more worthwhile. I began to understand the fundamental principles behind the methods like Scrum, XP and Lean, and learned a few new tricks that haven’t been used much before in SW engineering. Unfortunately Kent Beck hasn’t bothered to reference to the prior work and many of the Agile and Lean principles have been challenged and questioned, while their origins have remained unrecognized. For example the principle of quick iterations can be derived directly to the Goldratt’s production formula: the faster you can release the less Work-In-Progress or inventory you have. The less inventory not currently producing revenues, the lower are your capital costs in holding the inventory and the higher profit you have. Additionally you get also the feedback faster, which is a crucial meta-capability in the era of tightening competition between the Lean organizations (according to Robin Cooper: When Lean Enterprises Collide). Actually, when you think about it, the optimum iteration length or batch size is one or release cycle time close to zero, or in other words you could even use iterationless-process or a Drum-Buffer-Rope/JIT -pull-system as described by Goldratt.

A related building block of a JIT -production system are the Kanban -signal cards, that are closely analogous to the XP User Stories and the Scrum index cards. The idea is to pull the production from the customer/demand side by signaling the previous phases of the permit to produce new items, instead of producing documentation/code that might not be needed by anyone. The idealized model is the Just-in-Time -system, where a customer gets what he orders when he needs it, but nothing is produced in advance on stock. The magic in the system is the process capability, or the variance and the tolerance you need to have spare stock so that the sudden demand spikes won’t cause bottlenecks (or the increase in the cycle time lowering the critical throughput). According to the Deming’s Statistical Process Control -school, a process capability means how low variance the process has, or how low emergency stock you can keep without causing a bottleneck. A SW Engineering example of this is for example how many XP User Stories you have to produce before the beginning of an iteration, or can you also produce them Just-in-Time. At least on my XP team this reveals directly one bottleneck: we do have a queue of roughly half-foot of User Stories and the engineering team seems to be much slower to implement the stories than the customer is able to generate them. Also Goldratt notices that the easiest way to identify a bottleneck is by looking for the queue sizes, the longest queue is usually the bottleneck, especially if the queue seems to be always increasing.

From the process measurement point of view XP seems to draw the principles almost directly from the text book when introducing the Project Velocity -measurement – the only required measurement for a software team is its throughput, or how many user stories or story points they can release per iteration. Unfortunately Beck doesn’t explain very well why this measurement is the most important one, but you get it now: the most severe bottleneck in SW engineering is the coding, not the design – so you should first focus on getting your tput up. There is so much to do in improving the velocity that I don’t know yet where the next bottleck will be. I think it will take quite and much work before anyone can say that they have more designers than coders and are able to locate the next major software engineering bottleneck :) .

When you know the TQM -fundamentals you will also know the remedies for the performance and quality problems. The mission of the TQM -school (as first stated by Henry Ford) is to eliminate all waste. Somebody asked from me the what is the difference between the German quality and the Japanese quality? The difference is in the principle of eliminating the waste. When a car comes out from a German car factory, it goes first to a repair shop, and then to the customer. A Japanese car can be shipped directly to the customer. The difference is in how much scrap and rework you produce. As anyone in the software industry can easily recognize, the complete rewrites and bug fixes are the activities that most of the developers spend most of their days on producing also most of the cost. The Six Sigma -school goes even further and describes the Hidden Factories meaning all scraping and rework activities that are performed without appearing on the management reports. A good example is a bug that is detected and fixed immediately without entering it on the bug management system. This happens every day. One could think that a hidden rework factory is also a coding typo that requires use of backspace. According to the Japanese concept of Poka-Yoke, or error-proofing, the remedy is to use tools and practices that make it impossible to produce an error. For example you can use Eclipse that automatically displays the syntax errors, highlights the code, suggests for the possible (and correct) methods available, and autofills the brackets so that you can’t produce the elementary coding mistakes. Software engineering is full of social practices such as code reviews and pair programming that tries to further improve the quality by detecting the errors faster. However, practicing the Poka-Yoke (or the original Baka-Yoke = idiot proofing) will get you farther more quickly when you dismiss the possibility of producing errors in the first place! Yet more examples of the same principles include the Japanse concept of Jidoka- or a self-correcting machine that halts it’s operations and alerts a maintainer, when an unrecoverable error happens, and the Designed for Six Sigma, where product is designed by choosing components that are known to have low number of errors instead of coding all code from the scratch by self.

After introducing yourself on the key Lean/TQM concepts you should hopefully pretty quickly start to recognize from which more fundamental principle all the practices originate from, and in which situations they should and should not be used. I hope that with these few advices you can start to recognize where your bottlenecks currently are located in, and also get ideas how to improve your throughput and quality :) . I’ll be also writing more on the topic :)