A Software Developer's Perspective

Real world authentication and transaction protocol semantics

2012-01-09T04:10:00.000-08:00

A real world story explained in authentication and transaction protocol semantics to provide some insight into the relationships between real world information flow and failure demand that is so important when managing IT dependent service organizations.

A couple of years ago me and my family went on a trip to Mallorca. In the confusion of the arrival terminal baggage area we left one backpack on the pickup trail. The backpack contained artifacts of high value, but not critical to our stay. A human error started an obscure process enduring the our whole stay on the island.

Note: As most histories of failure demand this one started with the customer making a mistake. In systems thinking terms, this is probably so frequent in the charter business that it should be considered as mere variation.

On the bus to the hotel I discovered we where missing the backpack and asked the guide what to do. The guide told me to contact the guides located at the hotel as he was just having responsibility for transport to and from the airport.

Note: this must be happening all the time, why doesn't charter operators have procedures for handling this immediately? This is in effect a redirection to another service (endpoint), much like a http code 303 See other. This is also an example of standardized work that drives costs and complexity up.

I attended to the guides on the hotel on a daily basis. This resulted in using estimated 30 minutes per day in waiting and talking to the guides. Each day they promised to look into it, they contacted the airport, waited for response from the lost property department.

Note: I was polling the guides, which was in a constant loop answering me and polling the lost property department. This consumed resources and was of course a source of irritation to us.

In the day we where leaving, the guide having responsibility for the returning bus transport followed me to the lost property department of the airport. When asking for the backsack the clerk just shaked his head. Nothing like my backpack was found. I pulled my passport to eliminate any doubt of who I was, and provide correct information. The clerk quickly disappeared and returned with my sack. Huh? It appeared the guides was not informed of the required authentication to pick up left property.

Note: The airport clerks pretended it did not exist until I showed up in person and provided proper authentication. Much like a properly invented security measure in IT, but remember to inform your users (guides) about it. Failing to inform results in failure demand in the flow and bad service seen from the customer (which in this case will associate the travelling destination with bad service (cognitive availability).

Since the time before checkin was nearing fast I did not have time to check the contents of the sack before later. It appeared that mobile chargers and iPods where missing.

Note: Almost no security measure can stop unloyal employees from taking what they want.

After returning I claimed the loss to my credit card company. Travels payed with the card automatically has travel insurance. They presented a bureaucratic and cumbersome process, and in general did not cover lost property. Huh? So much for travel insurance. Our house contents (NO: innbo) insurance though covered lost property during travel. The process was very straightforward and used average prices on artifacts calculated from real prices. I got a security token that I could use to "buy" the artifacts again from their webpage or get money refund. -> Happy travelling insurance customer. I will stick with these guys for a while, as my availablity heuristic brain tells me they provides good service.

Note: providing a simple process with clear semantics reduces pain and resource consumption and great service.

Content and technology cycles

2012-01-06T14:22:00.000-08:00

Most content outlives any presentation and distribution technologies. Content valuable enough will be converted to new formats and prepared for distribution on new carriers. This is nothing new and has been ongoing since humans started to draw and write.

Content creation involves effort and possibly considerable amounts of economical investments. Professional content creators live off their work, and naturally wants to be paid for providing their works.

Somewhere along the road, movie-, music- and publisher industry acquired exclusive distribution rights. When they where distributing content in a physical form (in atoms), this added considerable value to the product. Distributors got involved in all kinds of coordination and promotion activities related to their exclusive rights. As long as the distributors controlled the technology this went along fine, but with the invention of the music cassette the foundation for a copy culture was laid. Content consumers wanted to have copies of music in their cars, walkmans and in the living room. The VCR allowed copying of films. A growing pirate industry, mainly based in Asia was founded, but the economical problems was limited as they had to move atoms around the globe and copied material was not more accessible than the originals.

For music and movies this led both to direct loss for content creators, but it also provided culture sharing. When more people heard new music or films, some bought originals while some copied. It will be impossible to know if the entertainment industry have lost or gained in this game. A copy today may result in future sales of releases of music and movies.

When computers moved into the home copying of software soon became common as the physical burden was shrinking. We started to move bits on physical carriers (diskettes). When PCs was introduced copying flourished. Whether this favored or hurt the software industry is not quite clear. Microsofts success with Office was laid with the massive distribution of Windows which was heavily copied and distributed amongst users. I would guess the software industry as a whole, especially platform vendors, have gained opportunities because of copying. This is not a defense of piracy, but my estimation of the outcome. The software industry has slowly adjusted to the new realities. Software can be bought and downloaded or provided as as service (SaaS). Open Source developers uses the moving of bits their full advantage.

In the late 90-ies the Internet was introduced for consumers. The Internet made moving atoms obsolete. Content could now be moved as pure bits globally, with the speed of the network. The Internet provides us with a completely new way of distributing and consuming content. It is also a foundation for collaboration, creativity and provides us with vast new opportunities for doing business.

The elimination of the need for moving atoms also reduces costs. There is little value in moving bits, and the concept of the moving and copying of bits breaks the limitations of physical distribution. It also breaks exclusivity as content can be made available for anyone instantly at a global scale. The entertainment industry seems not have understood or do not welcome this. They seem to be willing to go all the way in their efforts to stop the evolution. They lobby for and support legislations like SOPA, PIPA and EU Data Retention Directive.

Technology follows very rapid cycles of invention, while content has a completely different cycle. The most valuable content will inevitably be converted to new presentation- and carrier technologies. If the copyright owners does not do it, consumers will. And why should they not? If you have bought a product shouldn't you be allowed to use it on gadgets not yet invented? To me it is not crystal clear copying is theft. Copying is also culture sharing, and generates future sales. The Internet gives us great possibilities for sharing culture and this could be lost, or at least very restricted if the entertainment industry will get it's will with lawmakers.

So why would anyone tie exclusive distribution rights to specific distribution technologies? This gives distributors no incentives for adjusting to new realities and opportunities. These industries are now lobbying for draconic laws that is protecting their relatively outdated business models. They refuse to meet the demand of their customers and stubbornly gives Internet the blame for everything not going their way. The difference now is that consumers (and pirates) have all the tools and infrastructure to fill in the gaps. When the gap left open by the entertainment industry is wide enough and demand is high, massive piracy is the inevitable result.

Instead of exploiting vast opportunities for culture sharing and selling a lot more (for a lower price per item, reflecting reduced costs) they want laws that will seriously impede democratic freedoms hard won through recent centuries. The Internet has breathed new life to democracies (video of Al Gore talking about SOPA was quickly deleted from Youtube...). Whether the lawmakers does not understand this, or do not want more democracy will be mere speculation. It is probably a mix of incompetence and using piracy as an excuse for their own agenda.

Another consequence of the internet is that there is only one market: the global market. Trying to limit releases to restricted areas will fail and only cause problems. It has become de facto Cargo Culting in the media industry. In the digital distribution world there is no borders or physical restrictions that creates exclusivity. A better strategy will be global releases and making use of social media buzz to spread the word and attract consumers to buy content from copyright holders.

Hack your phone

2011-11-01T15:25:00.000-07:00

Recently my old HTC Hero started to get notoriously slow and frequently report "low on space". Even after removing all apps memory was still almost exhausted.The phone has been running on MoDoCo 2.1 since the last r5 release.

I could have just bought a new phone, which of course would be better by all means, but since the risk of bricking the phone was low (low probability and the impact of the risk was just to by a new one) I decided to find what the latest update around for an old phone like the Hero.

I found far more than I expected in Cyanogen Mod 7.1. Now I have a far newer OS than HTC has ever released for the phone, a lot of new functionality plus a much more stable and responsive phone. Extended standby time included.In addition to saving (or at least postponing replacing the Hero) it was a satisfying feeling to successfully tinker with the device.

When the risk is low the threshold for hacking your phone should be low. Just as people should know how to switch light bulbs and connect the pieces of a surround system, they should also have some knowledge about their phones and how to repair them. It is not hard, it is fun and you'll learn a lot.

My data

2011-10-14T11:48:00.000-07:00

We are doing it all wrong. Empowering people with technology is currently centered around the technology only. The technology is important of course, but it is totally useless without any data. Technology is just for playing with the data.

So do we empower the user with any control over data? Can we manage data digitally as in real life? Well of course not, digital information is about making data more fluid. But making data more fluid does not mean doing it to all data. The principles if privacy should be valid independent of format.

Private data should be as private as in a bank box. If the individual chooses to destroy the data, it is his or her choice.
Shared secrets should be as secret as though it was shared with real friends. Sometimes information will leak.
Published data should assimilate. As in real life. It would be impossible to control the flow
Observations can be recorded, but no personal information should be stored without consent

Given these simple rules, it should be the individual deciding which risks to take. Facebook, Google or governments should be in no position to decide what risks to take with any data.

That does not mean any data can be gathered by trusted parties, but what and how data is collection should be open. It must always be possible to opt out. That does not exclude companies and governments from monitoring activity. It should just be recorded as impersonal observations. As in real life.

Information industry battles

2011-08-29T08:19:00.000-07:00

Evidence that we are now witnessing one of the greatest battles in the information media industry is emergent. The giants of the industry is either actively pursuing ever more control or is being pushed to take defensive steps. Through the last century many similar battles has taken place: AT&T, the Hollywood Filmthrust and RCA vs Armstrong serving here as prominent examples.

The roles of the battle is not new: weak and unprepared governments, capitalism serving shareholders, and eager consumers. The goal of the current battle is to dominate so as to cut [exclusive] deals with content owners. As often seen before patents is used by the big guys to limit innovation from competitors, and push around those big enough to pose a competitive threat.

In earlier battles a very limited set of patents was used a weapons. Now the giants has to collect patent portfolios in order to gain sufficient control over/defense against competitors. A large number of consolidated patents is powerful when one tries to suffocate innovation and limit the innovative freedom of others.

Innovation has always been important in the Information media industry. Innovation could make the industry more or less self regulating. If a conglomerate/cartel manages to gain control over innovation that could be a threat and even ultimately replace them, they have also gained invulnerability. The loosers if the battle is the content consumers, that will have less choice. The free market can easily commit suicide, especially in the information industry.

The governments, and especially the USA, has much to easily given the giants the weapon they need: patents. It is maybe a bit counterintuitive, but patents is a construction for limiting further innovation. Governments are also generally weak at regulating the information media industry. This creates the opportunity to create an empire. The more powerful, the easier it is to get allies either by fear or business. But, there is a but, when governments has seen trough it's fingers of this battle yielding consolidated giants, they have also created a soft underbelly on those. The giants knows it, and parts of the governments knows it too. The parts of the governments that has understood, also know how to exploit it. Ultimately the information industry stiffens, only casting static shadows of its former dynamic nature. This is when capitalism stops working.

Recently the US and EU has implemented legislation that let them get access to the giants business records. In the name of the fight against terrorism and child porn, they have adopted draconian laws, that removes our digital privacy. This just get easier the bigger the giants gets, because the stakes get higher with size. No giant wants to be defeated up by an anti-trust case.

The current battle is more destructive than ever. The real stakeholders is not shareholders, but us. Our privacy is at stake. Information media industry collects private information and serves public information. It is a unbalanced game. We loose as consumers because without competetion, the giants will be lazy but almost impossible to replace. They have their patent portfolios. This equilibrium is exploited by our governments.

Patent dowry

2011-08-15T22:35:00.000-07:00

Patents, that are originally meant to promote innovation and protect investments, has become a strategical asset in the game of alliances. For Google it has become an urgent matter to strengthen the Android ecosystem with a patent portfolio, not innovations. I guess it is important to the Android partners that they are "protected" by patents. Not necessarily things they have invented. Just any patents that can be used in the war against the other players.

Google providing a patent portfolio is merely a dowry to make the Android ecosystem attractive and protect it's inhabitants. Innovation has become a minor variable in the equation. Innovating mostly pays off in lawsuits these days, because there is always someone who has bought or patented something the innovation resembles. The genius working alone that need to protect her or his ideas is a myth. Innovation happens in teams and cooperation with other entities, and is mostly empiric.

What really happens now is patents are collected in portfolios and presented as a deck, either to be attractive or frightening. Entities are forced to join conglomerates and consortiums in search of protection. If this game is allowed to proceed any longer it will be hard, and even impossible to enter the market. An idea that competes with the existing products will not have a chance. Innovation and the free market suffers. More or less willingly cartels are born through consolidation as a result of the patent wars.

Apple is already in bed with media industry, network providers, so we already have conglomerate of corporations controlling a large part of media consumption. Now with Google buying Motorola hardware a manufacturer is the same entity controlling the largest switch (search engine) of the Internet. These entities become very powerful. As long as they behave nicely, this is not a serious problem. But this system is vulnerable in two ways. First, how can such powerful entities restrict themselves so that this power is not misused? Secondly they have a soft underbelly, as they will probably be investigated in terms of antitrust. But governments have recently become very interested in how the Internet and electronic communications can be surveilled and even controlled.

Will governments regulate or exploit the opportunity? As long as the patent war proceeds, the conglomerates will not dissolve. It is their survival strategy. But it undermines the original design of the Internet with distributed control. It does not matter if the Internet is technically controlled in a distributed manner, when the information flow is centralized.

The situation will then resemble some of the pre antitrust cases in the information technology industry. But this time it is driven by patents.

To begin with, selling and buying patents should not be allowed as this fuels the patent war. But I guess it is much more complicated to fix this problem now than ever. Big patent owners will not like the idea of their patent portfolio, expensively procured, should only be used to protect innovative ideas for a short period of time.

PS! I listened to the JavaPosse #360 Newscast while writing this. It has some interesting points about these issues, recorded almost a week before Google buying Motorola, and as such is free of speculations over why.

Further reading
Patents, Schmatents!

Chaos is good - and why we should trust Google

2011-07-07T00:17:00.000-07:00

Google, the search engine, does not own content it directs users to, with some exceptions. I do not trust Google everything, and I do not say they always adheres to their own "do no evil" mantra. But when it comes to limiting the powers of traditional content owners it does a decent job. Google is the net neutrality's best bet. I will explain why I think this is so.

Google plays the same role as the phone catalog, but in a much broader sense. It helps users find what they look for. If the dispatcher(s) of the Internet is agnostic to newcomers and established content providers this will help innovation, education and general informing the global community. At the same time this degree of centralization represented by the mighty Google, is vulnerable. Misused or controlled neutral dispatching is broken and value limited.

A neutral dispatcher is important in ensuring net neutrality. Except from paid search rankings, Google's page ranking algorithm favors paths to content users prefers. I guess we are lucky to have Google. Google is no saint, and is not defending net neutrality out of pure values and standards. But it is important to their business and current position. They depend on users looking for content in a chaotic ever growing amount of content. Google creates value from chaos. Chaos is to their advantage. Content providers and owners, the kind that is lobbying against net neutrality, want control and order. E.g. Apple, married to AT&T in the US, wants you to find what you look for inside iTunes, and is does a heck of a good job providing a streamlined user experience.

Google, a strong player that does not have specific interests in providing owned content (Youtube beeing an important exception, but it is a free service and content is user provided) and with no network preferences they effectively have become kind of net neutrality guarantor. Because of their unique position in the information industry, it is their interest that the net is neutral towards all content providers.

As a consequence Google has never been closely related to content producers and network providers. Content providers even accuse Google of copyright infringements, even traffic is directed their way via the search engine.

With Android Google have become related to device manufacturers. But one should note that none of these, or at least to a small degree, has, produces or provides content. They compete on producing devices suitable for content consumption and to some degree content production.

Apple and Sony are examples of the opposite type of device manufacturers, with Sony Ericsson in a limbo position providing Android handsets. Sony Ericsson's Android handsets are typically more customized than other Android handsets. Apple is closely related to content providers through iTunes and AT&T on the network side for exclusive deal on the iPhone.

Content owners will want to shortcut the neutral dispatcher to gain advantages over other content providers. Ultimately main players in the information industry want to control distribution, and even consumer devices. By controlling the networks used for distribution and devices, they can direct, and even filter content available. This is called walled gardens, and what traditional content providers want. I interpret what happens in OECD as a content provider control coup towards network services. Content providers utilize their powers over governments to gain some (initial?) control over network services.

The architecture of the Internet, as Licklider and the other founders designed it premiered distributed media control. The design was in direct opposite of how AT&T and telephone networks was designed, and AT&T still struggles with this.

But Google can not alone defend against the forces trying to divide and conquer the Internet. Even Google will have to change business strategy if net neutrality is lost. Google is wholly dependant on the prosumer (producing consumers) and others betting on the open and neutral web (which of course is in their own interest too).

The World Wide Web and http protocol is used for providing, finding and consuming content. An open WWW is mutual dependant on net neutrality. Net neutrality is an important foundation for WWW as we know it, and at the same time helps defending against centralized control. The reason for this is the hyperlinking nature of WWW.

So what can net neutrality defenders do? The decisions needed is often counterintuitive, since you often will have to choose chaos over order and not first class content providers that only accepts exclusive deals. Prosumers must put an effort in putting all kind of common knowledge on the web under a sharing license e.g. Creative Commons to prohibit evasive copyrighting of stuff of interest to everyone. Digital tool makers must strive to give prosumers tools to mass produce good quality content. Prosumers should link to relevant content, that help glue the Internet. Network service providers must not make exclusive deals with content providers or let them to close to their operations.

Content makers should also consider what will serve them best in the long run: a close marriage with media conglomerates that was formed during the 90's or a model with room for all players in the content industries. In contrast to the job
market 20 years ago, the options are no much more dynamic. Lock-in is not a good thing when tools are democratized to a level where everybody can produce something (talent or not). A large part of the content industries will benefit from net neutrality in the long run, but established entities will often fight against it.

Governments must regulate to by intervening using antitrust laws, but when one studies the history of media industries it is revealed that they are slow movers and too vulnerable to lobbying. The book The Masterswitch, by Tim Wu, describes the phenomena of media- and content industries since the inception of telephony and is an important contribution in the net neutrality debate.

Can patents and intellectual property rights put a deadlock on the information society?

2010-08-24T12:34:00.000-07:00

What is information? Or what does information make us capable of doing? Information put into context is knowledge. Information exchange is the basis for empiric evolvment and great inventions. Without relatively free flow of information society can not evolve and prosper. Historically things would have turned out pretty differently of the alphabet was patented, or mathematics was protected as intellectual property.

During listening to a podcast interview with Robert Laughlin, that (I thought was only) about carbon future and climate, the talk also mentioned the topic of patents and intellectual property rights (at 50:35). Protection of informaton has restricted US to some extent to empirically evolve, and jobs is outsourced to e.g. Japan and now China. The patents is kept by American companies, but production is not in the US. It is however unclear how this affects employment and innovation in the long run, but there is a high probability that a connection is present. It started my thinking on how such protective measures affects our society. Laughlin mentions a book he has authored: Crime of Reason that rationalizes over this subject.

Just think how some cities and whole nations became recognized mariners in the era of sailships. By sharing knowledge, and demolishing the churchs false demagogy saying the world was flat, they conquered the earth. Little or no knowledge was patented before industrialization. At least not commodity knowledge.

The invention of the internet has let loose massive flows of information. Our society and daily lives is packed with technology. Information technology is ubiquitous and indespensable in the parts of the world calling themselves information societies. What disturbs me is that the tools we are so dependant on is illegal to tinker with to an increasing degree. Apple is the forefront of this development, but they are not alone. Given that a lot of smart people, buying products, sees ways to improve them it is a waste of talent not to let them. The knowledge of how the tools that we depend upon works should be available. Reverse engineering should not be am act of crime. The products themselves is just as valuable with the knowledge available, if not even more. When products can be extended in ways the manufacturer did not think of, the usefulness and usablility increases.

This is especially true for software, that increasingly becomes the inner workings of our tools. Did you know that the average car has software with over 10 million lines of code? How many knows how that code works, opposed to traditional home mechanic doing maintenance works on his own car? Recently it has been shown that wireless pressure sensors are vulnerable for malicous hacker attacks. Patents can not protect you from criminals, but people with good intentions (and I am fairly convinced they outnumber criminals) could reveals such things. The most capable could even provided fixes. Software should not be patented. The value is in goods that can be traded and valueadding services using the knowledge. Using information correctly is complex (instantiation of knowledge) and will always be in demand.

My point with arguing that knowledge about how our tools works is that this is how it has been most of the time during our civilization, and probably before that too. When there are too many patents and intellectual properties protected from reuse and tinkering, the information society may be deadlocked. If this is true, it is a slow process. It is like the story of boiled frogs, that do not recognize that their beein boiled when let into cold water slowly heating up.

The issues of protecting knowledge may eventually restrict desired and needed innovation. In the context of the interview with Laughlin, one can derive that it slows down or prohibit much needed concensus over what environmental challenges we are really facing and how they can be solved. To make it clear where I am going with this: Environmental challenges are global, the internet is made for global information exchange. The tools we use, that largely is the cause of (our perceived) environmental challenges, are protected from tinkering. Information protection and patents is not helping us in figuring what we have to do.

Has Oracle killed innovation on the Java platform

2010-08-14T01:25:00.000-07:00

Most of the innovation the software world is derivative works. The whole software innovation ecosystem is empiric, and new offsprings sees the daylight with knowledge originating from other successes and failures. Further, a lot of innovative products is based on commodity software, that saves innovators from the tedious and costly work of doing everything themselves. If these mechanisms breaks apart, the whole innovation ecosystem in the software world would crumble. I think this article explains how innovation happens today, and it lists som famous everyday innovations that where never patented. Things everyone of us uses almost everyday. Had they been patented, some things would be quite differently today.

So, Oracle sues Google over some patent infringments and IP rights. Because they own Java they can do just that. Google has been clever omitting these patents and IP rights and Sun did not see any interest in pursuing any possible infringements in court. I guess they saw Googles works as strengtening Javas overall position, even if the code was not portable, but knowledge is. And knowledge is very easily transferrable. Further, it can result in new offsprings and innovations.

The Java Virtual Machine is a commodity that a lot of business and open source projects relies upon. The desktop and server editions are open source under GPLv2. The mobile edition is not fully open source, and it is here that the Oracle lawyers (hyeanas are more appropriate me thinks) are seeing legal meat to dig into. How they axactly will argue is not yet revealed, but it could be they will argue lost business. I would say it is downright unethical to sue a another party on such basis. Just as Apples iPhone would not have had Androids market share, if it did not exist? There are no other real competitors to Apple just now, and is Oracle producing phones? Can they show a prototype? Can Oracle provide any proof that Java ME has lost any market share (as far as I know Java ME has not played any significant role in a market sense)? They are just hurting the JVM and Java language as a commodity by inserting insecurity and fright into the ecosystem. Maybe this will be the event that triggers completely new programming languages or strengthenes some new ones in the pipeline? Languages with absolutely no patent strings attached and potential misuse by the "owner".

The consequences could be devastating for innovation on the Java platform. Those who want to use Java the platform must from now on be extremely careful to not irritate Oracles lawyers. What does the Scala people think now? Will Oracle try to sue them for lost revenues on Java IDE's or do some ridicoulous changes to stop Scala?

In addition to damage innovation on the Java platform, this lawsuit will drain energy and time from managamenet at Google and Oracle. Who gains on that? Yes Apple and Microsoft. While Oracle bashes Google with stone age business models, competitors can exploit their distraction from the scene where innovation happen.

What feels so completly wrong with this lawsuit, apart from totally missing the point of the software industry works, is that Google has probably been one of the biggest contributors to the diffusion of Java language on the mobile platform, namele Android. Android provides no or little revenue for Google, but sees at as mere innovation and business platform that suits them. I think Oracle will have a hard time arguing for any economic losses as Google does not make money on Android directly and Oracle is certainly not in the ad-business.

Friday 13th , August, 2010 is a sad day in software history, and James Gosling did foresee it in the aquiring negotiations with Oracle. He must have felt like Albert in We, the drowned by Carsten Jensen. Albert can see war victims beeing killed in his dreams before it actually happen.

Update 18.08.2010
Charles Nutter has written a thorough blogpost on the issue.

Check your brakes

2010-05-12T12:25:00.000-07:00

This week I had several almost-accidents when riding my bike because my brakes was worn out. For every incident I got more careful and lowered my speed.

I changed my brakes, and I can tell you, it was a revelation. I could go much faster, and could stop almost instantly when required. The effects are only positive: I can go faster, safer and it is more fun. The safer part means I don't risk hurting myself and others. Good brakes kind of set you free.

During my first ride with the new brakes it struck me that this has analogies to many professional life as a programmer. When programming it is very easy to assume you have understood requirements, the technology I am using and all kinds of other assumptions.

Developers, projects and organizations should integrate feedback and brakes in their work. Good brakes is useless without proper signal to use them.

A brake in programming can be incarnated in several forms, which I will try to show here.

A programmer should listen to all feedback provided from compiler warnings, automatic- and peer code reviews. When struggling to name a software artifact, stop coding and take a break. Find a colleague that can act as a rubberduck or ask for advice. Do pair programming.

Unit tests provide concrete signal (when done adequately), and will in some ways act as a brake. They can let you refactor, delete and improve code more freely. A good test can also stop you from implement plain wrong functionality when you can not satisfy correct asserts in any way.

Likewise performance tests, security reviews, coding guidelines etc can act as signal to brake when things are going in the wrong direction. The more you can automate of these things, the faster it will let you go.

I think the principle of integrating feedback and brakes in all work done in an organization is valuable. For software projects frequent releases will provide feedback, and tell you occasionaly to clutch the brakes. Open discussions in the project can reveal bottlenecks and inadequate ways of doing things. No topic should be avoided, as avoidance can hide potential problems.

Requirements must be asserted too with prototypes and design can be explored and refined with CRC

A promising new technique that is beeing developed for decision making in software architecture can provide a much tighter feedback, brake and correction loop. I think the boardgame approach can be adopted for a lot of complex decision making processes.

A Policy Advisory Board (PAB) will be responsible for overseeing obedience of common rules and update outdated rules. At the organizational level the same yields as for projects regarding open discussion. Let people discuss freely, use wikis, and microblogging to unlock knowledge about inefficiencies.

The key observation of brakes, is not only the slowing down aspect. It is also about changing direction. Changing direction at high speeds may be impossible, risky or it has to be done with a large turn radius. Slowing down for a moment improves your quickness.

Now after a lot of talk about brakes it is important to state brakes are essentially a hook into work processes. These hooks should let people intervene to change direction. The brakes does not provide the required decisions that must be made, just an opportunity to make them before too much damage is done. They also provide excellent opportunities to learn. Failures is valuable learning and is an integral part of an learning organization. Learning is, amongst other things, effectively brake checking and the loop is complete.

Can tech obstruct your fundamentals?

2010-04-04T23:39:00.000-07:00

This Easter I came across this blog post about The Value Of Fundamentals (highly recommended reading), and it made me think about how and why we adopt new technology. (Maybe I am fond by this way of thinking because I was a martial arts practitioner myself, and highly respect them still.) In the adoption process the problem it is supposed to solve gets lost, and it is used everywhere as a one-solution-fits-all technology. Fundamentals is not as well understood as they should be in many professions, and software development is no exception.

Technology can prevent you from using your basic skills, as "advanced" technology may look like it can solve several of your problems at once. This is sometimes true, but more often just smoke and mirrors. In addition technology brings on a new set of problems (that you will try to solve with workarounds), and sometimes they become a hammer that you'll try to apply to any problem you stumble upon.

This occurs frequently in software development, but it happens elsewhere too, often with computer technology that is supposed to solve multiple problems. Way too often this obstructs the use of basic skills to solve problems. The challenges here can be mapped to other professions, e.g. physicians on large hospitals relying too much on all the available technology rather their basic skills.

Software development has always been ridden by the Silver Bullet Syndrome, and Silver bullets often infers technology that in effect makes us rely on factors out of control of the individual, project, and most important the stakeholders. But it is not only silver bullet technologies that can make you loose sight of real simple and elegant solutions to the most important problems you are trying to solve. Even wellproven and widely used technology may obstruct the view of the problems.

Here are some examples of categories of technologies that often obfuscates simple solutions:

Frameworks
Code generators
Integration technology, e.g. OR-mapping

Most of these adds complexity to the solutiuon (but promises an overall general complexity reduction), as they often are very general solutions trying to solve multiple problems. Solving complex problems by adding multiple frameworks to the solution make a very complex solution. This is where fundamentals becomes valuable, and decisions on whether to use a framework should be balanced with using basic skills.

When basic skills are forgotten or not practiced often you rely too much on technology to solve your problems. What happens then is that socalled advanced technology is applied to simple problems, that should be solved applying fundamentals.

What is really bad about using technology instead of basic skills in a software development context is that technology must be maintained throughout a products lifetime. When a technology vendor stops supporting a certain technology, all products relaying on it implicitly receive a death sentence. Products implemented mostly using basic skills has a better survival rate, as it often can be ported to new platforms where technologies on the legacy platform is unavailable. It can be very complex to update 3rd party frameworks if they are interdependant, or the code is invaded by the used technology. The maintenance cost can rise, and if ignored create serious technical debt.

Additionally well written software brings on best practices to newcomers reading the code, and when a skill/practice is questioned it triggers lot more valuable discussions than whether to throw out/introduce some technology. Evolving basic skills is a whole lot more valuable than decisions on the latest fad. Evolving basic programming skills empowers the individual , project, organization and profession. Technology brings on a substantial number of abbrevitions that is impossible to communication with users and stakeholders. Independant of what technologies are involved in your project, keep it out of non-technical discussions. Users don't care, and is certainly not impressed. Software that works, and that evolves with the users impresses. Wellcrafted software is easier to evolve in the long run, rather than fighting with framework/code generators.

So what fundamentals should all software developers master? I can easily come up with this shortlist, but it is not comprehensive or prioritized in any way:

Central design patterns
Know at least 2-3 programming languages, and 1-2 of these deeply and you should be acquainted with a scripting language.
Create readable and maintainable code
What is robust code?
Important concurrency concepts
Coupling theory

So this is a post that promotes the Not-invented-here syndrome? Not at all! Just let programming fundamentals weigh in heavier against e.g. frameworks and code generators. When cost and/or time constraints favor use of these, look at the source code (not only the documentation) and evaluate amongst other things:

it is well written
will not invade your code
not get in the way of creating elegant solutions to your problems.

As a pleasant sideeffect you might even learn a technique you are currently unaware of. When you understand how it is written it will be far easier to communicate with the vendor if changes must be made.

When you think about it, technologies come and go but the software industry in general changes at a much slower pace. It is time to shift this balance from praising the latest fads towards improving basic skills in the software development profession. This will make both the software itself, and the programmers, more valuable. Investing in fundamentals should also be incorporated in organizations' learning circle. Finally programming more consciusly with basic skills will contribute to less monocoltures in software, which in my opinion results healthier software.

Stone age business models

2010-03-03T22:53:00.000-08:00

It is sad to observere that patentbased lawsuits against competitors seems to be part of major mobile manufacturers business strategy these days. They have started to bang each other in the head using lawyers equipped with patents.

Nokia sues Apple, who sues HTC on UI technology. They've started to dig trenches instead of trying to beat the competion by innovating. High profile lawsuits like this require a lot of attention from the companies leaders, that distract them from making real business decisions. Especially Apple seems to have already forgotten about how fast they've recently grown in the smartphone market, only because they where innovative (and maybe already had a cool reputation). Well these times seems to have passed, and they've started to protect their innovations.

What Apple and Nokia is forgetting here is that this will force competion to do something uniquely new. Somewhere someone will come up with innovations making the iPhone look outdated. Innovation in this space often happens outside the big corporations, and this is becoming particularly true nowadays. The cloud offer startups vast computing resources, open source provide building blocks to start with and finally social media gives rapid feedback. No patent can protect an investment from this. See more on this here https://sites.google.com/a/webstep.no/openinnovation/Home/news-about-open-innovation/guykawasakioninnovationandthemythoflightningboltinspiration

I think patents in software is an anomaly that must be burried and forgotten. They can not help protect software investments, and only gives the patent grantees a false feeling of safety.

This week a new way of doing UI, Skinput, was presented, and that from the patent borg in Redmond, or at least a Microsoft driven University. I guess this will be patented too since it is a Microsoft-led innovation, but nevertheless it comes from a "unexpected" source, as Microsoft has lately been accused of not innovating much

Update: Very interesting podcast on the Techrepublic on this subject

Update 12.03.2010: Some very interesting and relevant blogposts:
The New Paradigm of Advantage and Jonathan Schwartz on Patent Litigation

Major- and Minor Tyrannies in software

2010-02-20T23:43:00.000-08:00

Serviceorientated Architecture (SOA) can enable/support redesign of business processes that can help organizations tap more of their potential or even providing uniquely new products and services.

But there are som pitfalls that might show up later in the process as obstacles, caused by lack of/forced- or accidental decisions. The pitfalls I will elaborate on here are those that dictate projects to use inapropriate technologies for the problem they are supposed to solve. I call these majority and minority software architecture tyrannies. Software architects must spot these and handle them properly.

Majorities
Majorities often force inappopriate solutions on other projects in an organization, disquising this as standard solutions to be used. This eliminates good and qualified decisions in projects that is doing something different than previous project. Doing something different is the norm in software projects, as it often is part of renewal/change in business/technology.

Such standards is often invasive and impossible, or at least very hard, to change later.
Majority tyrannies must be met with knowledge about better alternatives, and how they can contribute to better supporting requirements and business strategy. In cases where standard noninvasive technology is forced, an isolation layer can be introduced to prevent unwanted dependency diffusion into architecture and code.

Standards often appears as a relieve, liberating the project architect from making decisions. Not staying alert can prove fatal to the project later on.

Minorities
Survivor projects clinging to old or inappropriate technology can prevent others from moving on to better or more suitable technologies. Measures dependant projects can take is to create a Integration Anti Corruption Layer (orginal definition from DDD Anti Corruption) layer, so it will be easier to replace later.

A subset of minority tyranny cases is lack of proper versioning of dependencies, making releases of different projects interdependant. In SOA this is amplified and has become a runtime challenge, as opposed to earlier where this was primarily a build time problem. Several versions of shared services (and components) must be supported simultaneously to enable independent and smooth releases. In SOA the most flexible way of version handling is using the Evolving Endpoint pattern.

Consequences and how to deal with this
The concequence of making wrong/forcing/avoiding architecture decisions is high complexity in release management and ineffective software for supporting business processes. Since both organizations and software technology changes continously, decisions can not be be written on stone tablets. Previous decisions must be challenged, and discarded as they are
a) proven wrong
b) outdated
c) proved unecessary

This is part of SOA Governance and must be handled by a Center Of Excellence or Policy Advisory Board

Especially majorites often lead to Architectural monocultures, which is bad both seen from innovation (evolution) and security point of views. These are strong motivators for evaluating and make architectural decisions based on business requirements rather than what has worked before.

This post may appear as a anti standard manifest, but that is not my intention. Good standards have been reevaluated many many times, and survived these evaluations. This "process" will run the test-of-time on standards and give feedback for refinement. I think it can be viewed as variation of natural selection, where the fittest survive. Natural selection, by the way does, not apply in monocultures. Monocultures can produce odd mutations and eventually they collapse.

On a per service this manifests itself as providing multiple endpoints, that makes it as accesible, usable and flexible as possible. A service consumed by many clients gives it a strong position in the organization(s) using it, and thus it may itself become a well proven standard.

Excel and grep is the logfile analyzers best friend

2009-11-27T08:56:00.000-08:00

Lately I have been analyzing some logfiles for invocation times of remote services. Luckily the most problematic part of the system logs time consumption together with a remote system identifier and some other stuff.

Some years ago I was working on SmartLearn, implementing analytics for learning accountables. Through that work I got to know Excels Pivoting capabilities, but back then I used Microsoft Analysis Services for creating the Pivot tables. I have also been a user of Linux-, Unix- and Cygwin's sharp commandline tools for a long time. Seperately I know strength of the tools from both of these worlds, but I did not recognize how I could use them together.

One of my project peers showed me how Excel could be used to extract data from flat files and present it as Pivot tables in Excel with very few steps. The key premise is that the interesting dimensions of the Pivot-table is logged on the same lines as the interesting numbers/text. The technnique described here will let you visualize counting of things. Even without any numbers the frequence of things can be very interesting. Most system log files contains a timestamp, and this can be combined with almost anything *. Using *nix-commandline tools it is of course possible to extract whatever information you like from flat files.

17:48:05,168 DEBUG Task1:29 - end call, Task0 duration=401

To extract only these lines I use Cygwin's grep command like this:

grep Task.*duration <logfilename> > extractedlog.txt

From Excel open the extracted log file. Excel will recognize the file as a textfile that you might want to split in columns. Choose between splitting at fixed positions and characters. Spaces can be a viable option in some cases.

When you have imported the file it may be necessary to split columns manually using the Text To Columns tool in Data toolbar.

Now you must insert a row at the top of the dataset, and add header names of the interesting columns. Cut & paste the interesting columns so they are adjacent to each other.

Now select the Insert tool in Excel, and Pivot Table (leftmost button in my installtion of Excel 2007 under Insert)

Choose either Table or Chart, and select the interesting fact and dimension columns from the spreadsheat. When you click OK, you can start dragging & dropping columns into Axes, Values and Legend containers on the right side.

You can now twist the logdata as you want, and find relations between them you would spend enormous amount of time finding manually in the log file. Some simple examples:

To create the dataset I used this Java code that produces differentiated exection trends in a number of threads (which is not meant as a educational example of how to write multithreaded code):

package com.webstep.logfilegenerator;

import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;

public class LogGenerator {

/**
* @param args
*/
public static void main(String[] args) {
  ExecutorService exSvc =  Executors.newFixedThreadPool(3);

  List<Future> futures = new ArrayList<Future>();
  final Random r = new Random();
  for (int i = 0; i < 10; i++) {
      futures.add(exSvc.submit(new Task1("Task"+i, r.nextInt(i+1*500))));
  }  
  try {
      for (Future f : futures) {
          System.out.println(f.get(10000, TimeUnit.MILLISECONDS));
      }

  } catch (InterruptedException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
  } catch (ExecutionException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
  } catch (TimeoutException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
  } finally {
      exSvc.shutdownNow();
  }
}
}

import java.util.Random;
import java.util.concurrent.Callable;

import org.apache.log4j.Logger;


public class Task1 implements Callable {
private static Logger log = Logger.getLogger(Task1.class.getName());
private final static Random r = new Random();
private final int maxTimeout;
private final String name;

public Task1(final String name, final int maxTimeout) {
  super();  
  this.maxTimeout = maxTimeout;
  this.name = name;
  log.debug(name + " maxtimeout=" + maxTimeout);
}


public Object call() throws Exception {
  while (true) {
      log.debug("start call " + name);

      long start = System.currentTimeMillis();
      Thread.sleep(r.nextInt(maxTimeout));
      log.debug("end call, " + name + " duration=" + (System.currentTimeMillis() - start));
  }
    
}
}

Some sample output from this code:

17:47:56,798 DEBUG Task1:19 - Task0 maxtimeout=98
17:47:56,803 DEBUG Task1:19 - Task1 maxtimeout=12
17:47:56,803 DEBUG Task1:25 - start call Task0
17:47:56,804 DEBUG Task1:19 - Task2 maxtimeout=17
17:47:56,804 DEBUG Task1:25 - start call Task1
17:47:56,806 DEBUG Task1:19 - Task3 maxtimeout=51
17:47:56,807 DEBUG Task1:19 - Task4 maxtimeout=92
17:47:56,807 DEBUG Task1:19 - Task5 maxtimeout=47
17:47:56,808 DEBUG Task1:19 - Task6 maxtimeout=8
17:47:56,809 DEBUG Task1:19 - Task7 maxtimeout=87
17:47:56,809 DEBUG Task1:19 - Task8 maxtimeout=66
17:47:56,810 DEBUG Task1:19 - Task9 maxtimeout=39
17:47:56,808 DEBUG Task1:25 - start call Task2
17:47:56,876 DEBUG Task1:29 - end call, Task2 duration=65
17:47:56,878 DEBUG Task1:25 - start call Task2
17:47:57,030 DEBUG Task1:29 - end call, Task2 duration=151
17:47:57,031 DEBUG Task1:25 - start call Task2
17:47:57,174 DEBUG Task1:29 - end call, Task0 duration=369
17:47:57,175 DEBUG Task1:25 - start call Task0
17:47:57,296 DEBUG Task1:29 - end call, Task1 duration=490
17:47:57,297 DEBUG Task1:25 - start call Task1
17:47:57,323 DEBUG Task1:29 - end call, Task0 duration=146
17:47:57,324 DEBUG Task1:25 - start call Task0
17:47:57,390 DEBUG Task1:29 - end call, Task1 duration=91
17:47:57,391 DEBUG Task1:25 - start call Task1
17:47:57,448 DEBUG Task1:29 - end call, Task2 duration=415
17:47:57,449 DEBUG Task1:25 - start call Task2
17:47:57,549 DEBUG Task1:29 - end call, Task1 duration=157
17:47:57,550 DEBUG Task1:25 - start call Task1
17:47:57,729 DEBUG Task1:29 - end call, Task2 duration=278
17:47:57,730 DEBUG Task1:25 - start call Task2
17:47:57,802 DEBUG Task1:29 - end call, Task0 duration=477
17:47:57,803 DEBUG Task1:25 - start call Task0
17:47:57,886 DEBUG Task1:29 - end call, Task1 duration=334
17:47:57,887 DEBUG Task1:25 - start call Task1
17:47:58,048 DEBUG Task1:29 - end call, Task2 duration=316
17:47:58,048 DEBUG Task1:25 - start call Task2
17:47:58,068 DEBUG Task1:29 - end call, Task0 duration=264
17:47:58,069 DEBUG Task1:25 - start call Task0
17:47:58,087 DEBUG Task1:29 - end call, Task1 duration=199
17:47:58,092 DEBUG Task1:25 - start call Task1
17:47:58,098 DEBUG Task1:29 - end call, Task1 duration=5
17:47:58,098 DEBUG Task1:25 - start call Task1
17:47:58,099 DEBUG Task1:29 - end call, Task2 duration=48
17:47:58,099 DEBUG Task1:25 - start call Task2
17:47:58,115 DEBUG Task1:29 - end call, Task1 duration=17
17:47:58,115 DEBUG Task1:25 - start call Task1
17:47:58,245 DEBUG Task1:29 - end call, Task2 duration=146
17:47:58,245 DEBUG Task1:25 - start call Task2
17:47:58,339 DEBUG Task1:29 - end call, Task0 duration=267
17:47:58,340 DEBUG Task1:25 - start call Task0
17:47:58,398 DEBUG Task1:29 - end call, Task0 duration=57
17:47:58,399 DEBUG Task1:25 - start call Task0
17:47:58,412 DEBUG Task1:29 - end call, Task0 duration=12
17:47:58,412 DEBUG Task1:25 - start call Task0
17:47:58,542 DEBUG Task1:29 - end call, Task1 duration=427
17:47:58,543 DEBUG Task1:25 - start call Task1
17:47:58,573 DEBUG Task1:29 - end call, Task2 duration=326
17:47:58,574 DEBUG Task1:25 - start call Task2
17:47:58,741 DEBUG Task1:29 - end call, Task1 duration=197
17:47:58,742 DEBUG Task1:25 - start call Task1
17:47:58,783 DEBUG Task1:29 - end call, Task0 duration=369
17:47:58,784 DEBUG Task1:25 - start call Task0
17:47:58,835 DEBUG Task1:29 - end call, Task2 duration=260
17:47:58,835 DEBUG Task1:25 - start call Task2
17:47:58,873 DEBUG Task1:29 - end call, Task0 duration=89
17:47:58,873 DEBUG Task1:25 - start call Task0
17:47:58,923 DEBUG Task1:29 - end call, Task2 duration=87
17:47:58,923 DEBUG Task1:25 - start call Task2
17:47:58,942 DEBUG Task1:29 - end call, Task0 duration=67
17:47:58,942 DEBUG Task1:25 - start call Task0
17:47:59,084 DEBUG Task1:29 - end call, Task1 duration=342
17:47:59,084 DEBUG Task1:25 - start call Task1
17:47:59,091 DEBUG Task1:29 - end call, Task2 duration=166
17:47:59,091 DEBUG Task1:25 - start call Task2
17:47:59,115 DEBUG Task1:29 - end call, Task0 duration=172
17:47:59,115 DEBUG Task1:25 - start call Task0
17:47:59,348 DEBUG Task1:29 - end call, Task2 duration=257
17:47:59,348 DEBUG Task1:25 - start call Task2
17:47:59,473 DEBUG Task1:29 - end call, Task2 duration=124
17:47:59,474 DEBUG Task1:25 - start call Task2
17:47:59,505 DEBUG Task1:29 - end call, Task0 duration=389
17:47:59,505 DEBUG Task1:25 - start call Task0
17:47:59,506 DEBUG Task1:29 - end call, Task2 duration=31
17:47:59,507 DEBUG Task1:25 - start call Task2
17:47:59,534 DEBUG Task1:29 - end call, Task1 duration=450
17:47:59,535 DEBUG Task1:25 - start call Task1
17:47:59,579 DEBUG Task1:29 - end call, Task1 duration=44
17:47:59,580 DEBUG Task1:25 - start call Task1
17:47:59,629 DEBUG Task1:29 - end call, Task2 duration=121
17:47:59,630 DEBUG Task1:25 - start call Task2
17:47:59,667 DEBUG Task1:29 - end call, Task0 duration=160
17:47:59,668 DEBUG Task1:25 - start call Task0
17:47:59,747 DEBUG Task1:29 - end call, Task1 duration=167
17:47:59,748 DEBUG Task1:25 - start call Task1
17:47:59,852 DEBUG Task1:29 - end call, Task2 duration=221
17:47:59,852 DEBUG Task1:25 - start call Task2
17:48:00,153 DEBUG Task1:29 - end call, Task0 duration=485
17:48:00,154 DEBUG Task1:25 - start call Task0
17:48:00,156 DEBUG Task1:29 - end call, Task1 duration=408
17:48:00,156 DEBUG Task1:25 - start call Task1
17:48:00,185 DEBUG Task1:29 - end call, Task2 duration=332
17:48:00,185 DEBUG Task1:25 - start call Task2
17:48:00,316 DEBUG Task1:29 - end call, Task2 duration=130
17:48:00,317 DEBUG Task1:25 - start call Task2
17:48:00,435 DEBUG Task1:29 - end call, Task2 duration=117
17:48:00,436 DEBUG Task1:25 - start call Task2
17:48:00,542 DEBUG Task1:29 - end call, Task1 duration=385
17:48:00,543 DEBUG Task1:25 - start call Task1
17:48:00,629 DEBUG Task1:29 - end call, Task0 duration=474
17:48:00,631 DEBUG Task1:25 - start call Task0
17:48:00,720 DEBUG Task1:29 - end call, Task2 duration=283
17:48:00,721 DEBUG Task1:25 - start call Task2
17:48:00,809 DEBUG Task1:29 - end call, Task1 duration=266
17:48:00,810 DEBUG Task1:25 - start call Task1
17:48:00,886 DEBUG Task1:29 - end call, Task1 duration=74
17:48:00,887 DEBUG Task1:25 - start call Task1
17:48:00,908 DEBUG Task1:29 - end call, Task0 duration=276
17:48:00,909 DEBUG Task1:25 - start call Task0
17:48:01,023 DEBUG Task1:29 - end call, Task2 duration=302
17:48:01,023 DEBUG Task1:25 - start call Task2
17:48:01,140 DEBUG Task1:29 - end call, Task0 duration=230
17:48:01,140 DEBUG Task1:25 - start call Task0
17:48:01,349 DEBUG Task1:29 - end call, Task1 duration=462
17:48:01,350 DEBUG Task1:25 - start call Task1
17:48:01,378 DEBUG Task1:29 - end call, Task2 duration=355
17:48:01,378 DEBUG Task1:25 - start call Task2
17:48:01,518 DEBUG Task1:29 - end call, Task2 duration=139
17:48:01,518 DEBUG Task1:25 - start call Task2
17:48:01,599 DEBUG Task1:29 - end call, Task1 duration=249
17:48:01,599 DEBUG Task1:25 - start call Task1
17:48:01,603 DEBUG Task1:29 - end call, Task0 duration=461
17:48:01,603 DEBUG Task1:25 - start call Task0
17:48:01,939 DEBUG Task1:29 - end call, Task0 duration=335
17:48:01,940 DEBUG Task1:25 - start call Task0
17:48:01,997 DEBUG Task1:29 - end call, Task2 duration=476
17:48:01,998 DEBUG Task1:25 - start call Task2
17:48:02,014 DEBUG Task1:29 - end call, Task1 duration=414
17:48:02,015 DEBUG Task1:25 - start call Task1
17:48:02,146 DEBUG Task1:29 - end call, Task1 duration=131
17:48:02,147 DEBUG Task1:25 - start call Task1
17:48:02,261 DEBUG Task1:29 - end call, Task0 duration=320
17:48:02,262 DEBUG Task1:25 - start call Task0
17:48:02,343 DEBUG Task1:29 - end call, Task2 duration=342
17:48:02,343 DEBUG Task1:25 - start call Task2
17:48:02,378 DEBUG Task1:29 - end call, Task1 duration=230
17:48:02,378 DEBUG Task1:25 - start call Task1
17:48:02,520 DEBUG Task1:29 - end call, Task0 duration=257
17:48:02,520 DEBUG Task1:25 - start call Task0
17:48:02,597 DEBUG Task1:29 - end call, Task2 duration=253
17:48:02,598 DEBUG Task1:25 - start call Task2
17:48:02,734 DEBUG Task1:29 - end call, Task1 duration=355
17:48:02,735 DEBUG Task1:25 - start call Task1
17:48:02,786 DEBUG Task1:29 - end call, Task0 duration=265
17:48:02,787 DEBUG Task1:25 - start call Task0
17:48:02,959 DEBUG Task1:29 - end call, Task2 duration=361
17:48:02,959 DEBUG Task1:25 - start call Task2
17:48:03,017 DEBUG Task1:29 - end call, Task0 duration=228
17:48:03,018 DEBUG Task1:25 - start call Task0
17:48:03,027 DEBUG Task1:29 - end call, Task2 duration=66
17:48:03,030 DEBUG Task1:25 - start call Task2
17:48:03,214 DEBUG Task1:29 - end call, Task1 duration=478
17:48:03,215 DEBUG Task1:25 - start call Task1
17:48:03,299 DEBUG Task1:29 - end call, Task2 duration=268
17:48:03,300 DEBUG Task1:25 - start call Task2
17:48:03,305 DEBUG Task1:29 - end call, Task1 duration=90
17:48:03,306 DEBUG Task1:25 - start call Task1
17:48:03,324 DEBUG Task1:29 - end call, Task1 duration=17
17:48:03,325 DEBUG Task1:25 - start call Task1
17:48:03,345 DEBUG Task1:29 - end call, Task2 duration=44
17:48:03,345 DEBUG Task1:25 - start call Task2
17:48:03,456 DEBUG Task1:29 - end call, Task1 duration=130
17:48:03,456 DEBUG Task1:25 - start call Task1
17:48:03,484 DEBUG Task1:29 - end call, Task2 duration=138
17:48:03,484 DEBUG Task1:25 - start call Task2
17:48:03,491 DEBUG Task1:29 - end call, Task2 duration=6
17:48:03,492 DEBUG Task1:25 - start call Task2
17:48:03,496 DEBUG Task1:29 - end call, Task0 duration=477
17:48:03,496 DEBUG Task1:25 - start call Task0
17:48:03,696 DEBUG Task1:29 - end call, Task0 duration=199
17:48:03,696 DEBUG Task1:25 - start call Task0
17:48:03,796 DEBUG Task1:29 - end call, Task1 duration=340
17:48:03,809 DEBUG Task1:25 - start call Task1
17:48:03,841 DEBUG Task1:29 - end call, Task2 duration=348
17:48:03,841 DEBUG Task1:25 - start call Task2
17:48:03,845 DEBUG Task1:29 - end call, Task1 duration=35
17:48:03,847 DEBUG Task1:25 - start call Task1
17:48:03,876 DEBUG Task1:29 - end call, Task0 duration=179
17:48:03,876 DEBUG Task1:25 - start call Task0
17:48:03,952 DEBUG Task1:29 - end call, Task2 duration=111
17:48:03,952 DEBUG Task1:25 - start call Task2
17:48:03,999 DEBUG Task1:29 - end call, Task0 duration=123
17:48:04,000 DEBUG Task1:25 - start call Task0
17:48:04,220 DEBUG Task1:29 - end call, Task1 duration=372
17:48:04,221 DEBUG Task1:25 - start call Task1
17:48:04,348 DEBUG Task1:29 - end call, Task2 duration=395
17:48:04,349 DEBUG Task1:25 - start call Task2
17:48:04,385 DEBUG Task1:29 - end call, Task0 duration=384
17:48:04,392 DEBUG Task1:25 - start call Task0
17:48:04,633 DEBUG Task1:29 - end call, Task2 duration=284
17:48:04,634 DEBUG Task1:25 - start call Task2
17:48:04,678 DEBUG Task1:29 - end call, Task1 duration=457
17:48:04,679 DEBUG Task1:25 - start call Task1
17:48:04,704 DEBUG Task1:29 - end call, Task1 duration=25
17:48:04,704 DEBUG Task1:25 - start call Task1
17:48:04,766 DEBUG Task1:29 - end call, Task0 duration=373
17:48:04,766 DEBUG Task1:25 - start call Task0
17:48:04,912 DEBUG Task1:29 - end call, Task2 duration=277
17:48:04,912 DEBUG Task1:25 - start call Task2
17:48:04,928 DEBUG Task1:29 - end call, Task2 duration=15
17:48:04,928 DEBUG Task1:25 - start call Task2
17:48:04,943 DEBUG Task1:29 - end call, Task1 duration=238
17:48:04,944 DEBUG Task1:25 - start call Task1
17:48:05,072 DEBUG Task1:29 - end call, Task2 duration=143
17:48:05,072 DEBUG Task1:25 - start call Task2
17:48:05,132 DEBUG Task1:29 - end call, Task2 duration=59
17:48:05,132 DEBUG Task1:25 - start call Task2
17:48:05,135 DEBUG Task1:29 - end call, Task1 duration=191
17:48:05,135 DEBUG Task1:25 - start call Task1
17:48:05,168 DEBUG Task1:29 - end call, Task0 duration=401
17:48:05,168 DEBUG Task1:25 - start call Task0
17:48:05,239 DEBUG Task1:29 - end call, Task1 duration=103
17:48:05,240 DEBUG Task1:25 - start call Task1
17:48:05,357 DEBUG Task1:29 - end call, Task2 duration=224
17:48:05,357 DEBUG Task1:25 - start call Task2
17:48:05,526 DEBUG Task1:29 - end call, Task1 duration=286
17:48:05,526 DEBUG Task1:25 - start call Task1
17:48:05,555 DEBUG Task1:29 - end call, Task2 duration=197
17:48:05,555 DEBUG Task1:25 - start call Task2
17:48:05,600 DEBUG Task1:29 - end call, Task2 duration=44
17:48:05,600 DEBUG Task1:25 - start call Task2
17:48:05,616 DEBUG Task1:29 - end call, Task0 duration=447
17:48:05,616 DEBUG Task1:25 - start call Task0
17:48:05,650 DEBUG Task1:29 - end call, Task2 duration=49
17:48:05,652 DEBUG Task1:25 - start call Task2
17:48:05,772 DEBUG Task1:29 - end call, Task2 duration=120
17:48:05,773 DEBUG Task1:25 - start call Task2
17:48:05,775 DEBUG Task1:29 - end call, Task1 duration=248
17:48:05,775 DEBUG Task1:25 - start call Task1
17:48:05,782 DEBUG Task1:29 - end call, Task2 duration=9
17:48:05,782 DEBUG Task1:25 - start call Task2
17:48:05,865 DEBUG Task1:29 - end call, Task2 duration=83
17:48:05,866 DEBUG Task1:25 - start call Task2
17:48:05,870 DEBUG Task1:29 - end call, Task2 duration=4
17:48:05,870 DEBUG Task1:25 - start call Task2
17:48:05,878 DEBUG Task1:29 - end call, Task0 duration=261
17:48:05,879 DEBUG Task1:25 - start call Task0
17:48:06,105 DEBUG Task1:29 - end call, Task2 duration=234
17:48:06,105 DEBUG Task1:25 - start call Task2
17:48:06,239 DEBUG Task1:29 - end call, Task0 duration=360
17:48:06,239 DEBUG Task1:25 - start call Task0
17:48:06,258 DEBUG Task1:29 - end call, Task2 duration=152
17:48:06,258 DEBUG Task1:25 - start call Task2
17:48:06,260 DEBUG Task1:29 - end call, Task1 duration=485
17:48:06,260 DEBUG Task1:25 - start call Task1
17:48:06,452 DEBUG Task1:29 - end call, Task0 duration=213
17:48:06,452 DEBUG Task1:25 - start call Task0
17:48:06,586 DEBUG Task1:29 - end call, Task1 duration=326
17:48:06,586 DEBUG Task1:25 - start call Task1
17:48:06,619 DEBUG Task1:29 - end call, Task0 duration=166
17:48:06,620 DEBUG Task1:25 - start call Task0
17:48:06,672 DEBUG Task1:29 - end call, Task0 duration=52
17:48:06,672 DEBUG Task1:25 - start call Task0
17:48:06,747 DEBUG Task1:29 - end call, Task2 duration=489
17:48:06,747 DEBUG Task1:25 - start call Task2
17:48:06,792 DEBUG Task1:29 - end call, Task1 duration=205
17:48:06,792 DEBUG Task1:25 - start call Task1

* Updated 2009-11-29

Open Source will never get out of stock

2009-10-31T00:41:00.000-07:00

It is like running out of light or wind. Software can be copied as many times as needed. Why haven't Microsoft and Apple understood this yet?

Some random examples of Windows 7 beeing out of stock
http://crave.cnet.co.uk/software/0,39029471,49303067,00.htm
http://www.digi.no/827047/windows-7-utsolgt-i-sverige

How can software be sold out? The only commodity invlolved is the software to be copied. To produce a physical copy a USB drive or recordable CD or DVD must be used in the process. But basically software can be copied between computers over the network. The required production equipment is computers, electric power and a network. These resources are abundant in the enterprise and in the average european/american home.

It seems like it is mostly pirates and the Open Source movement that have understood this distribution model. Commercial vendors seems to try to ignore or deny it. That is too bad, because they could profit from the extremely low distribution costs.

Pirates exploits the low distribution costs to distribute goods illegally
Open Source have relied on this distribution model for a long time. This has resulted in a very broad open source software usage, and distribution costs is approximately zero.

While Open Source and pirated digitial goods has very different usage profiles, they share the common properties of extremely effective and almost non-existing distribution costs. Open Source is mainly used by software developers, and consumers download software and digital media content for free.

In the last couple of years a new distribution patterns has gained foothold, beyond the most simple forms: SaaS. SaaS leverages software that can be used as-is instantly. Since there is no significant download and installation, the costs/effort required from the end user is even lower than with Open Source. For example the latest release of Ubuntu aims to provide a OS as a service by leveraging Amazon EC2 compatible images.

Software vendors, and other digital content providers, should free themselves from physical medias. Soon physical medias will be of interest of the entusiast. The rest of us will prefer instant gratification over the physical media.

Last weekend I was in a record shop where they of course played some music. I was very surprised when I saw the PC with Spotify was used instead of a CD player. That says it all about the state of physical distribution of digital goods.

Monocultures are unhealthy - even in software

2009-10-21T23:04:00.000-07:00

Making the assumption that every project should be based on the same software stack, is just another variation over Silver Bullet.

In the software world the temptation to default to an architeture that has worked before is unhealthy. The result is almost always a constant struggle for the project to overcome limitations and find workarounds for architetural monstrosities.

Monocultures fosters very little learning in the organisation and leads to forcing inappropiate solutions to problems. According to epidemology theories monocultures is very vulnerable and is unable to evolve to tackle changing environment. Thinking of epidemology naturally leads to thinking about security issues as well. Monocultures is only able to resist specific types of threats, and given that the threats is certainly evolving at a blazing speed there is a obvious need to have variation.

In the longer term Monoculture will hinder innovation, and this can disastrous for an organization whose business model is e.g. developing and selling software.

But of course not all variations will be good, and should be dismissed. When experimenting with a technology new to the organization or a given project, do it in the small before going full scale. The opposite of genetical monoculture is diversity. Healthy software architecture in an organisation is probably best grown in an evolutionary way, allowing varations, promote the things that work well (but not restrict to only that). More important: things that do not work well must die.

All systems should be architected with the "right set" of technologies for the problem it is suppose to solve. One should start with what you know for sure, and make as few assumptions about the future as possible. The Cantara Software foundation has a wiki discussing these issues among other things, and I will try to post more in this topic there.

Yesterday I came across a blog post in Cutter Consortium, about uncertainty in a leadership perspective. This applies to software architects too. This should be in the architect's mind when evolving the software arcitecture for the organization.

DDD and Ubiquitous language

2009-09-09T10:47:00.000-07:00

A recurring subject on several sessions on the 1st day of Javazone 09 was Domain Driven Design and Ubiquitous Language. I do agree with those believes it is important to establish a language that can be used by domain experts and developers to ensure consistency.

Just before I had to leave I attended the DDD panel discussion where it was discussed whether this language should be used in the code as well. Well that is completely natural for English speaking developers. For developers in countries where English is not a native language, this raises some questions:

The code can not be maintained by people not familiar with the language the code is written in. This is very relevant in outsourcing.
It will pose problems when companies from different countries merges and must update the code and integrate to reflect the new situation.
Code that will or can be exposed as open source or in company partnerships will be of less value if it is written in non English

My personal opinion is that Norwegian (my native language) does not look well in code. It feels unnatural.

Is it responsible to write code in a non English language? I think this should be discussed with the stakeholders. On the other hand when there is a language impedance, how do you deal with it? Is it the developers responsibility?

I wanted to ask the panel these questions, but had to leave to catch a bus.

Updated: Corrected Domain Driven Development to Domain Driven design and added useful links.

Do you master social media?

2009-08-26T22:47:00.000-07:00

This post is almost "off topic", and does not follow up on the series I announced in my previous post. Think I've forgot what I was thinking of then, but I'll surely get back to software and architecture later.

The background for this post is that I can not help myself getting frustrated over uninformed criticism over social media that a lot of old hat media seems to love these days.

Social media is, by many, viewed as a tornado of (useless/disconnected) information. On the other hand it is easy to find examples of constructive use of social media, that provides extremely useful information and knowledge to it's readers. For some examples look at my blogger following on the side of my blog. Sometimes I get irritated when people has strong negative opinions against social media (especially microblogging), stating it makes us dumber and unable to concentrate anymore. Well that is up to the individual to decide what to make of it, and decide to pick up a book instead of e.g. "hypertweeting".

I have stumbled upon extremely useful knowledge on Twitter, blogs and more professional web publishers, reading and discussing it through social media's unique features. My opinion is that social media is a whole lot better way of staying informed and increasing your knowledge than watching TV. Well at least a lot of what is distributed via TV is not exactly what tickles your brain to think.

To be really interesting in the social media space you will have to consume considerable amount of information, and possess the capability to transform and magnify this information to your readers. This requires deep concentration and is exactly how it always has been. Just think of ancient Greek philosophers. Some have large information processing capability and others not. Given that some has better capabilities than others, everyone can improve their skills. Social media provides the best training ground ever for improving the skill to express your knowledge. As Chris Anderson says in The Long Tail, the tools for expressing yourself in public writing has been democratized.

In my view expressing and discussing your knowledge is probably the most valuable thing a person can do for oneself and society in general. The ability to express your knowledge through writing is more important than ever. I am a strong believer of knowledge is key to improve individuals and mankind. Each and everyone should acquire the knowledge they need to make informed decisions for themselves or in a position as a decision maker for others such as leaders and politicians. With social media it is even possible to discuss things on a completely democratic way before things is realized.

Twitter - an untested spacecraft

2009-06-13T22:57:00.000-07:00

Oh no, not a another post analyzing the growth of Twitter I hear you say. But here I will only use it to exemplify how software architects can and should prepare for rapid growth of services that are or will become an integral part of the social Web.

This post is first in a series of posts I am currently thinking of, on Software Architecture for the Social Web. Througout the post I will use Service Categorization defined at Cantara Wiki.

It will be irresponsible to implement a software system with all necessary investments for future scaling to large amounts of traffic. It is equally irresponsible to not architect it to let it grow so that it can handle large loads.

A software system "exposed" to the Social Web can have millions of potential users. That will apply to all applications that everyone can subscribe to. With the possibilities of OpenId and Facebook identity provisioning a subscription is not many clicks away. Who, when and how many will try the service depends on it's attractiveness and word-of-mouth on the web.

Twitter is exceptional in several ways: It as had an enormous growth, and has a lot of growing pains. It is unusual for a H2A service that have been that unreliable to sustain such a remarkable growth. Launching Twitter must have been like launching a space craft that had not been tested for the stress it has been exposed to. Astronauts and ground crew has only 1 chance of success: there will be no repair or improvements after launch. Luckily software can be repaired after launch, but it cannot be assumed it will be easy and with little impact on it's users without planning for it.

I don't know exactly how Twitter's architecture have evolved from it's initial release, but there must be at least 3 key takeaways for architects (seen from the outside):

It has always been easy to integrate with Twitter through it's API (A2A services), leveraging valueadded H2A services and clients
It has improved to handle the current load without breaking the API and the clients.
Except from a specific feature that was removed (causing an uproar), no functionality has been broken as I know of

These points is important to all software architects, but on the social web failing to fulfill them will certainly be a disaster. I guess not all services will experience such faithful users as Twitter, that has kept coming back despite instability at times. Twitter is probably highly valued among a lot of users, is free to use and can not be held economical- or legally responsible for any loss of data or unavailabilty.

When architecting a service that have to fulfill a SLA, or is paid for, some level of stability will be assumed. Failing to do that might have economical- , legal- and market implications. This applies to all services, and especially when offering commercial services exposed to the social web, some key architecture aspects most be thorughly planned. To number a few important ones:

All services (that is all types described on Cantara Wiki, but especially H2A and A2A) has to be orthogonal. This ensures least painful replacements when they become bottlenecks or otherwise insufficent.
Clean interfaces and clearly defined responsibilites at all levels of the architecture.
Conscious use of tradeoffs to deliver early but still keeping the architecture agile to let it evolve.
Measuring of performance and resource usage must be built-in to identify bottlenecks early.

These things will be essential for all projects aiming at the social web as its user base. It will be hard to predict adoption rates, and it will seldom be economically feasable to make it extremely scaleable before launch. Most projects must release early to start the revenue stream and get feedback from users on what to implement next.

What can Open do for you?

2009-05-16T04:59:00.000-07:00

Yesterday Totto noticed me of this Gartner blog about how CIO's should answer when asked about what IT contributes to the enterprise. Mark McDonald suggest that the answer should be how IT contributes to the core business model. This way it can be shed some light on how business models and IT can both be improved to be better positioned against the competition.

Another blog post that caught my interest yesterday was Matt Asay's
Cloud computing: A natural conclusion of open source? Here Asay is writing about Tim O'Reilly's prediction that Cloud Computing will make Open Source licenses irrelevant. A bit far from what my post is about, but it stresses that new software will be consuming/provider of services. In order for new software based on the SOA/Cloud Computing paradigms, open data formats and open standards will be quite important. Open Source is a natural way of implementing both data formats and standards, and hopefully this is going to happen. Even inside enterprises these thoughts should be considered, so it will be possible to combine the result of different development efforts to create even more valuable software to support the core business model.

2 days ago, I fell in love with this blogpost from Rickard Öberg about quickness. Although this post is about quickness at the individual developer and team level, it applies to organizations as well. To quickly reposition against the competition and market demand, sharp services exposing open data formats and beeing based on open standards will be crucial.

The CIO should understand what Open can do for the core business model, as this will be utterly important as more software is SOA based, running in the cloud. The ability to move quickly has been important at all times, but in some ways the changes in paradigms is accelerating and it will be more visible when you does not posess this ability.

Yesterday, Now and Tomorrow

2009-03-25T23:21:00.000-07:00

Designing software is a continuous task, requiring enduring effort and attention.
I strongly believe that the Code is the design. All efforts leading to the delivered code is just preparations for creating the design that gets released.

During the lifetime of a software product better and less good decisions are made along the way. Some decisions are made consciously and others more accidentally. After reading and discussing Technical Debt with my colleagues lately, I have come to think that is other economical terms that also fits on design decisions like investing. Investments can pay off, not paying off and not loosing, and loosing. What you do is make a bet. The greater the investment, the more certain you should be that is the right thing to do.

Ward Cunningham coined the Technical debt phrase. Martin Fowler and Steve McConnell has elaborated on it. This post tries to tie the investing term into software design. I really hope that by using these terms it is possible to create a software design meta language that is better understood by non technical decision makers.

Yesterday's decisions
Bad design from the past create technical debt. This debt should be downpayed (not just paying interests). It has to be a prioritized objective in every project to resolve this into better solutions.

Now
Sometimes there is a need to do something to the design just get the next release out the door. These decisions typically creates debt as a rule of thumb. Make these investments as small as possible. Update: Got a useful comment from @javatotto, about that the effect of these kind of design decisions should be isolated, so as few dependencies to them are created from elsewhere. Great comment that could be topic for the next post ;-)

Tomorrow
Investing for tomorrow is speculating. This is up front design, and often considered bad. This is especially true for projects that need the be able to change direction fast. It also requires an effort not helping you get the next release out of the door. You might even invest in something you will not need, which will represent a complete loss.

Small investments seldom make big trouble and can not represent big losses.

Big investments for tomorrow needs to be watched closely as they can potentially ruin the project. Maybe the worst thing about big investments are that they are extremely hard to dispose when they prove to be a loss. There is a cognitive attachment (ownership) that hinders replacing them with better solutions. The processes around big investments should be open, so that all people having a interests in it can have influence on how it develops. As I am a eager proponent for Open Innovation, I think especially these kind of investments can profit from developed as open source or at least open as possible.

Not paying down on the existing debt will ultimately be the terminator for a project. Final. I have seen some correlation between big investments and not paying down the technical debt in most projects I have participated.
Big investments for the future has potential to steal energy and focus (the projects short term capital) from things that should be fixed. Often it is the most talented developers involved in both activities, and they seldom manage to to both simultaneously. The worst case is when doing the wrong investment. That makes this even worse, as the debt grows dangerously fast.

My Web 2.0 stack

2009-02-21T07:16:00.000-08:00

I have earlier blogged about how Firefox is becoming my OS. Since then I've employed several Web 2.0 services, and Firefox is again a key tool for getting everything running smoothly. While some of the services will work in the exact same way with other browser it is the complete experience I am talking about here.

First there is plugins for sharing links. Since last post on this subject I've added Feedly.com, Friendfeed.com, Flickr.com, Youtube.com, Digg.com and Facebook.com (did not use it for sharing that way before) to the list. To make sharing as easy as possible the Shareoholic plugin comes in handy. This plugin can share links on almost every service that exist.

For now Twitterfox had to go, as I've started to use Tweetdeck, and if they run in parallell my Twitter API limit is reached pretty fast.

It has been a while since I started using Feedly, but it is only recently I have discovered what a great service it really is. Together with Google Reader, Feedburner and Friendfeed, it just makes everything very very easy and smoothly. Feedly instantly shows me how many have digged a page, how many have shared on Friendfeed. I can directly post to Twitter, and probably a lot of other things I have not discovered yet. And did I mention it is a great and innnovative news service. You've really have some aha's to discover if you haven't tried it yet, but be sure to check out Google Reader first, since it is tightly integrated with that service.

Because everything works across all OSes, and it is really fast to set up on a new computer (just bought a new home pc so its verified), I can move across machines and stay informed all the time. It was Adobe Air + Tweetdeck that was the most time consuming thing to install, because it is a fat client application. I find it pretty useful, therefore I bother installing it on multiple machines.

2 Icons, 2 movements

2009-02-11T22:15:00.000-08:00

Last night a thought struck me: there is a striking similarity in how the open source- and snowboarding movements has evolved. Both has been led by strong iconic persons namely Terje Håkonsen and Richard Stallman. They are both controverse untouchables (as in the movie with Kevin Costner), they do and say exactly what they believe in. Their influence has been very important in establishing these movements that is not controlled by any commercial interests, although none of them would have made it this far with commercial support.

Håkonsen is reknown for boycotting the IOC and FIS and olympic qualification for halfpipe when it was introduced in the omlympics. Stallman is so reknowned for his controversery that a lot of people thinks he is just a childish troublemaker.

The reason these movements have become so powerful, even in a commercial sense, is that they consists of large crowds of people. They have to some varying degrees developed ethics and morale, and those not conforming are effectively kind of excluded or frozen out. The crowds is not driven by commercial interests, but it would be false to state that is not commercial interests involved. Snowboarding is not for economically faint hearted, and software development is seldom gratis.

An example showing the power of the Open Source movement: Try to imagine the WWW without the Apache Http Server. It is undeniable the very reason the acronym http is known probably by half or more of the world population. It has lost "market shares", but it is the Apache http Server that has made the widespread deployment of the WWW possible. It has alwasy been on the frontiers of the WWW, and I guess it still is. In addition most application servers has an Apache http Server in front of them to handle caching and serving static content and many other tasks. I think the success for Linux in the server room is largely because of Apache http Server. The Apache Foundation has long been sponsored by IBM.

The Open Source movement has laid a foundation for business, information spreading and social interaction over the internet that would not have taken place or would have looked completely different without it. In fact, I think Open Source has been genuinely good for the internet and computing in general, lowering the bar for adoption. It makes it possible for everyone the freedom of speach if they want to.

Open Source is again showing it's strength in times where financial crisis otherwise might have strangled innovation. Instead innovation on the internet is flourishing, largely powered by Open Source. I think now the movement has become strong enough to transform the software business. I do not dare predict how it will transform it but I think it will be good.

During the US election campaign in 2008 i think I overheard a statement made by John McCain , when he started to realize he was not winning the election: The man (Barack Obama) is a movement. You can't stop a movement.

BTW: This weekend I will do practice som freedom of speech here: http://twitter.com/OfficalWRC

Let Open Source set you free and be unique

2009-02-08T22:01:00.000-08:00

A bit of a bold heading, you would say? Well that may depend on your attitude against and knowledge about Open Source. My opinion is that when you master a set of open source products, they will let you develop more freely. This is probably not that different from proprietary products, but with Open Source you can get deep knowledge very fast. Chances are that there are many developers "out there" having expert knowledge about most widely used Open Source products.

This post focuses on these advantages of Open Source:

Competence can be inhouse and not just on a support line
Defines what is more or less standards of how software us built today
No up front costs and procurement process

Now I will try to explain what I mean by stating this.

Inhouse competence
By hiring or employing the adequate Open Source competence software companies can get a head start on a project. The good thing about this is that the competence is not on the other side a phone or just being an email address. When challenges are met or there is a problem in production, the competence is right there with you.

Software standards
There is a lack of defined standards on how software should be designed. By design I mean what this article by Jack W. Reeves says. I do not doubt the value of visual software design, but this is just a vehicle for making the correct code design.
A lot of Open Source products has made such a strong foothold in software development that they more or less dominate in some categories of applications, e.g. the Spring Framework . Spring framework is very flexible, but by following the recommended conventions you will probably run into less trouble (not that impedes more trouble than other frameworks). That said, Spring is not very intrusive, so it should be pretty simple to adjust to conventions that is often also common sense.
When you adhere to the common sense conventions it mostly makes the software produced easier to use with other components and frameworks. It is necessary though to have the necessary competence when decisions is made along the way, just as you would with proprietary products. The difference, again, is that you do not have to get it from the vendors support or experts.

No procurement process
There is no costs, or at least they are very low, so getting permission to use Open Source is not an issue to discuss with the financial department. It is necessary to evaluate different products, also proprietary when they exists, for the project. Until now commercial products has often won these evaluations, but mainstream development is taking a new direction. I am not surprised that is happening right now, as the financial crisis raises questions about costly software.

An important consequence pointed out in the Agile Executive post, is that the software lives before the eyes of the user because with Open Source you can move as freely as the available competence is able to produce new features. This is very attractive, as the users keep coming back for exploring new functionality.

Another important point here is that when new technology surfaces, there is easier to switch when there has not been investments in commercial products. Software with a good design is flexible and makes as few assumptions of it's surroundings as possible.

Be free and unique
Now to the point of this post! Open Source let you, with the right competence, focus on the things you are good at, and deliver faster than with proprietary products. The business value or end user experience is what counts. Users mostly does not care how software is designed, but they value good software.
When you can focus on this, chances are that the product will be better and the users more happy. Unhappy users can express them self freely and quickly on the internet, especially after the advent of Web 2.0. A product must often rely on the word of mouth. With Twitter and the like being adopted at a blazing speed, word of mouth is spreading very fast.

Examples that this is true is flourishing:

Facebook
Twitter
Android
Google Chrome, the browser no one expected and got Google a lot of positive attention

BTW this post is written in Firefox, utilizing the Delicious plugin all the time.

Web 2.0: Use , create and evaluate it

2009-01-27T22:57:00.000-08:00

Yesterday Andrew McAfee wrote a excellent post on what characterizes Enterprise Web 2.0 software. Key points from the blog is that Enterprise Web 2.0 software must be:

Freeform
Frictionless
Emerging

Use it
All software team members should use Enterprise Web 2.0 tools to learn about the possibilities it opens up and to improve the processes. Especially Web 2.0 software is good at capturing the messy parts legacy processes and other tools has failed to encompass. Most Agile projects has already adopted wikis, for supporting the process, to scale beyond the ideal number of project members. Today many projects are geographically distributed or members telecommute. Here the social media part of Web 2.0 will be valuable.

Create it
During the whole process of developing software the characteristics of Web 2.0 software described by McAfee must be consulted.
These abilities must be ubiquitous in the software design and architecture both when developing brand new Web 2.0 software but also when extending legacy software with modern features.

These characteristics will probably evolve so stay tuned the Web 2.0-sphere.
Most developers will not create all features of a software product in the projects they are participating in. This will certainly be true for Web 2.0 software since a lot of Web 2.0 services are present already on the internet. They will only integrate with these software to add value to the service or vice versa. Software products may even thrive on existing social software with large user bases. By authenticating with e.g. OpenId or Facebook Connect you instantly get access to a large mass of users. Facebook users will probably soon be so spoiled that they may even try new services that does not support Facebook Connect. It also useful to look for open dataformats, such as those described at Microformats.

Evaluate it

Developers and others must evaluate existing software Web 2.0 products and services that they have to integrate with. Short time to release and economical constraints forces reuse of existing software/services. Especially reuse (social media) that has large user bases can be fruitful and be a critical success factor.
The characteristics McAfee describes should be used when evaluating Web 2.0 software.

The Economy and Innovation

2009-01-15T03:56:00.000-08:00

Nowadays a lot of things seems to happen that will influence on enterprises and individuals. The economy takes unpredictable turns, and innovation may suffer from capital starvation because more restrictive and careful financial investment policies.

Innovation inevitably comes with risk, and a lot of investors will be much more careful in respect to if and where they place their money. Careful investors may perform a much more thorough investigation of potential investments on beforehand than before.

My guess is that projects that has a solid innovation model, based on Enterprise Web 2.0, will have an advantage. This innovation model should include collaboration from all stakeholders, including its users. What this means to each and every project will vary, and Enterprise Web 2.0 must be adapted to the target organization(s) and individuals and vice versa.

Open Innovation can be seen as Enterprise Web 2.0 in an innovation context. Open Innovation recognizes that it is not affordable or rational that enterprises invents products solely from it's own research. Knowledge is distributed and must be gathered into collaboration.

I think Open Innovation can be taken one step further fully utilizing Web 2.0 software, with Social Media and Wikinomics to enable Mass Collaboration at a large scale. A popular term for this is Crowdsourcing.

Enterprise Web 2.0 connects knowledgeable people in new ways through social media, and collaboration through wikis is far more effective than email. Wikis is there for every interested person to read and contribute, and is not limited to mailing list. Further everyone is informed at the same time, at least all those subscribing to feeds from the wiki.

The community that arises around an idea or project will be a part of the backing capital, that nurtures it with capital, energy and direction. When there are several strong (economical or knowledgable) stakeholders involved, risk is spread.

So how can a software development enterprise start thriving from Open Innovation? Guess there will be no easy answer for that. In general I think it will be good to start in the small, opening up extension points that business partners can be interested in collaborating and provide value added services. Discussions and documentation must be located at a wiki, and people should be able to get acquainted with each other through some social media.

Opening up extension points in a software product, clearly puts some requirements on its architecture. To begin with, explore microformats and authentication solutions like OpenId, then consider if other formats must be invented.

I have established a Google Site, a subsite of my employer Webstep's site, for this subject. Please read more at http://sites.google.com/a/webstep.no/openinnovation/

Firefox is becoming my OS

2008-11-05T08:37:00.000-08:00

There has been predictions for some time now, that the browser will become the OS meaning that importance, and even existence of, an OS will be less important.
E.g. here is a blog that discusses the issue.

I haven't come that far yet, as a developer I still need applications installed locally that depends on having a OS like Windows or Linux. But there has been some real changes in which applications I use, and how I access them.

First of all, blogging, social networking and getting informed is new activities that I now do regularly in the browser. My social networking activities has become quite varied and wide, so a lot of time is spent using the browser for these activities. All mail accounts is GMail, even for job purposes. My employer, Webstep is using Google Sites for exchanging all types of information, and it is accomplished completely using a browser. All this, and most of the time I do not even have to push the save-button. I wonder how many work hours has been lost because Microsoft products does not have auto save...

Firefox also has a lot of useful addons available. Ubiquity is definitely one of them, as I can do a lot of things with the content on the web pages I visit. I also use TwitterFox a lot, and GMail Notifier. The delicious plugin is used heavily and has solved my bookmark synchronization need once and for all!

Most of my tools I use at work is OS independent (Java), so it does not really matter if I use Windows, Linux or OSX. It's just a matter of personal- and project related preferences. The application that most often forcing me to use Windows nowadays is iTunes (no I will not provide a link!), and that is not freely. For my N95 I by music from 7Digital and Telenor Music Store which both works excellently in the browser across OSes. We have an iPod and to sync it, I find it easiest to just do what Mr Jobs tell me to do.

As a contradiction, I am now starting to use Maven more seriously on my daily work. Earlier the projects I have been working could not use Maven because of several reasons. With the m2Eclipse+Nexus plugin installed there will be less need to use the browser to find OSS components. Probably the largest step forward in improvement of my development tools in a long time. It really makes handling dependencies between components easier and a whole precise.

Finding Open Source code and components with Unit Tests

2008-10-28T11:08:00.000-07:00

Yesterday I read an interesting article in IEEE Software about Code Conjurer Eclipse plugin and Merobase.

Merobase is a code search engine supporting some quite interesting features, among them querying by writing Unit tests. According to the IEEE Article, candidates matching the interface expected by the implementing class (under test) is actually tested with your test on the Merobase server.

Classes found that successfully fulfills the expectations in the test is shown in the search result.

Another interesting feature, is that results can be filtered in Code Conjurer showing only matches satisfying compliant licensing regimes that you specify.

So far I have only tested the web browser search interface, with some varying quality in the result. Anyhow this is a very exciting idea, and is probably something companies with large code bases would want to have in house.

This service has a real potential to reduce the "reinvent-the-wheel" problem, and improve collaboration inside and between organizations. In addition it can even get more people writing good unit tests.

Just one more thought: Why hasn't Google already done this with Google Code Search?

Nowhere to hide (from) the Evil State

2008-09-23T23:13:00.000-07:00

There has been some warnings sometime already from gurus, bloggers and article writers that we will have to get a lot more careful regarding writing code that is thread-safe. Most of the time programmers do it right, not necessarily because they have understood the whole problem, but because the default singleton behavior of the Spring framework forces them to.

On the other hand some old sins comes into the spotlight these days. Code that was written when developers only had single cpu-machines, and the multi-cpu machines really was not fast enough to uncover all potential problems.

Now developers has discovered what they can do with multi-cores and start writing new code utilizing them. The problem is that they reuse older code that was not developed with this in mind.
Most code handles data belonging to different users correctly. The problems often arises when code is executed multithreaded on behalf of a single user. Then there is nowhere to safely store state without lock-management anymore. Not even the database safe. It is no longer safe to assume that even a database only used by 1 system that rows "belonging" to a specific user can be updated without proper lock handling.

The old saying that "State is Evil" (wondering who said that?) has got a new renaissance. All objects belonging to the HttpSession must be managed properly for instance. I can see this can become challenging in a JSF application if you have the model is JSF managed beans. JSF does not let you (easily) replace the value of a managed bean. Guess there is some API's, but are they thread safe?
This means that if multiple threads is given access to controller- or model beans there must be proper locking on updates to the model.

JavaZone 08 meltdown

2008-09-18T13:01:00.000-07:00

I think this year's JavaZone was a bit varied considering quality and diversity of sessions, but all in all it was great. Is I posted yesterday some of them really inspired me. Todays session was not that "wooow", but I enjoyed some of them quite a bit. The BBS folks seem to really have gotten testing right in their Continuous Integration loop, and Unclebobs refresher on clean code was enjoyable.

I was a bit puzzled though by the first session today by Jason van Zyl where he showed a lot of bugs in his latest build of m2eclipse. The parts that did work I liked a lot, so I hope they get these things fixed. The parts about Nexus went better.

This year the JavaZone organizers introduced the use of Twitter and Hashtags for live feedback from the audience and speakers. It seemed like the number of twitters increased substantially from day 1 to day 2. Guess some got introduced to Twitter at Club Zone. Anyway it is cool that JavaZone introduced microblogging to the attendees, and I hope this will catch on in Norway and Java community especially.

Proprietary innovation - where has it gone?

2008-09-17T10:10:00.000-07:00

Not that I miss it, but I still wonder after the first day of JavaZone 08. Most, if not all, sessions presenting something innovative was open source based projects or developers presenting new ideas.
I think this generally is a healthy development but is commercial Java Vendors now only implementing JSR's, and rely on the open source community to come up with real innovations. Well, partly, they are sponsoring and involving in open source projects, but my observations from the first day of JavaZone is still that the really interesting things going on is driven by independent developers. Adobe Flex seems to be the only exception, but that is a good one though as it is very innovative it is production ready and pretty useful too.

2 sessions really got me today: Richard Öbergs Qi4j and Anders Norås' presentation of Better Domain Driven Design
Both gave new impulses and ideas on how Domain Driven Design can be further improved. As always Ôberg delivers a bit of controversy and noncomprimising radically new thoughts. Qi4j offers a Composite Object framework utilizing Annotations and Dynamic proxies that enables composition of objects (not classes).
The framework is initself a very good idea, but unfortunalety Öberg could not recommend using it in conjunction with frameworks like Spring although some connectors exists.
Any framework must have (real good) capabilities of coexistence and beeing able to cooperate with other frameworks in all but very small systems.
I am also a little worried over deep stacks of dynamic proxies, which will have performance implications and possible obfuscating stacktraces (although Öberg said these would be removed from Exception stack traces), but these things tends to pop out anyway.

Anyway I will keep an eye on what's happening with Qi4j but I am afraid the gap from the currently widely used frameworks is too big.

Anders Norås presented a similar, but much simpler approach to the same problem domain. By using Guice and ordinary Java Generics he demonstrated much of the same capabilities of extracting non-domain-code into separate classes. I would give his approach a better chance of getting foothold, at least in the near future. On the other hand this approach did not provide as many goodies as Qi4j, e.g. the visualizer and elegant UnitOfWork management.

Both approaches provides innovating ideas to the Java community, they are typesafe and supports refactoring. If Arjen Poutsma, that also speaks at the confence, together with the other two could come up with intelligent ways of making these ideas and technologies available together they will provide even more value. It is seldom a choice not to include one or more of other widely used frameworks in projects today.

I had to go home, before the real fun starts, as I must be able to deliver the car for repair tomorrow at 06:30 before I return to day 2 of JavaZone 08. No Cafè Con Bar for me this year.

com.sun.rowset.CachedRowSet pains

2008-05-20T10:52:00.000-07:00

Yesterday I came across a nasty NullPointerException in the com.sun.rowset.CachedRowSetImpl class. The application has used rowset.jar for over a year now when using Spring JdbcTemplate, and it has been working as it should. Suddenly when deploying to WebSphere 6.1 we got


java.lang.NullPointerException
        at java.util.Properties$LineReader.readLine(Properties.java:404)
        at java.util.Properties.load(Properties.java:332)
        at java.util.PropertyResourceBundle.(PropertyResourceBundle.java:110)
        at com.sun.rowset.JdbcRowSetResourceBundle.(Unknown Source)
        at com.sun.rowset.JdbcRowSetResourceBundle.getJdbcRowSetResourceBundle(Unknown Source)
        at com.sun.rowset.CachedRowSetImpl.(Unknown Source)
        at org.springframework.jdbc.core.SqlRowSetResultSetExtractor.newCachedRowSet(SqlRowSetResultSetExtractor.java:83)
        at org.springframework.jdbc.core.SqlRowSetResultSetExtractor.createSqlRowSet(SqlRowSetResultSetExtractor.java:65)
        at org.springframework.jdbc.core.SqlRowSetResultSetExtractor.extractData(SqlRowSetResultSetExtractor.java:49)
        at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:393)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:342)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:403)
        at org.springframework.jdbc.core.JdbcTemplate.queryForRowSet(JdbcTemplate.java:446)

A year ago the project was using Sun JDK 1.4, which does not include the com.sun.rowset package. Sometime last year we upgraded to Sun JDK 1.5, and no one thought of removing the rowset.jar from the runtime classpath. Each developer runs the application on JBoss, and the rowset.jar has not caused any trouble here. The application is deployed as an exploded EAR in development.

When packaging for WebSphere the rowset.jar was put in /lib directory in the EAR, and the developers assumed that the IBM JDK 1.5 did not contain the com.sun.rowset package. Shmack, there the NullPointerException reared its ugly head.

Inside rowset.jar, there is a property file called com.sun.rowset.RowSetResourceBundle.properties that is loaded through this little nasty piece of code in com.sun.rowset.JdbcRowSetResourceBundle:


    private JdbcRowSetResourceBundle()
        throws IOException
    {
        Locale locale = Locale.getDefault();
        String s = locale.getCountry();
        if(!s.equals("") && !s.equals("US"))
            propResBundle = new PropertyResourceBundle(Thread.currentThread().getContextClassLoader().getResourceAsStream("com/sun/rowset/RowSetResourceBundle_" + s + "." + "properties"));
        else
            propResBundle = new PropertyResourceBundle(Thread.currentThread().getContextClassLoader().getResourceAsStream("com/sun/rowset/RowSetResourceBundle.properties"));
    }

I am living in Norway, so the code will actually be trying to load com.sun.rowset.RowSetResourceBundle_NO.properties which of course is not provided in the rowset.jar file. I really hope the developer at Sun who made this has a real good explanation for writing this.

Well, today we discovered that the IBM JDK 1.5 does provide the com.sun.rowset package, and the whole problem was resolved by removing the whole jar from the project. There is nothing like the good feeling when getting rid of som crappy code!

One of my peer project developers found this post that sat me off in the right direction. Especially the link to Ted Newards excellent ClassForName.pdf

BTW: http://del.icio.us/dagblakstad proved invaluable while resolving the problem and writing this post.

I'm now a Nokia Sportstracker Fan

2008-05-07T23:29:00.001-07:00

The Nokia Sportstracker is one of the coolest thing around of you have Nokia GPS enabled phone.

Have a look at the combining with the Sports Tracker widget everyone can see that i'm not in very good shape.

In order to view the Nseries Widget you need JavaScript and Flash Player 9+ support

Black Hat Antipattern

2007-09-26T00:26:00.000-07:00

On the way to work yesterday, I sat on the bus and was thinking of problems related to 3rd party components with source code you have no access to. Software modules that is not shipped with source code is often called black box components. This has historically been viewed as a good thing, but with the advent of Open Source, the advantage of having the software available has become evident. Before the weekend I had been trying to integrate with a backend system by using a inhouse developed library, which of course I did not have the source code for.
After struggling a long time with the library, which kept complaining in the log about a NullPointerException and a follow-up about a missing protocol I had to surrender. The library kept it's secrets tightly to it's chest.

While I was thinking about this, thoughts about magicians that pulls rabitts out of black hats came to my mind. The analogy is that the library pulls some configuration out of the Black Hat, but I did not have a clue of what to put in it to satisfy the library. The configuration file contained a property name protocol but it did not make any difference what value I supplied for it.

When I arrived at work a coworker was struggling with an exact analogy. He was using a low-level library (a sort of Data Access component), which was dependent on the HTTP session via JSF FacesContext. The is no recipee for success when using the library in a Axis 1.4 Webservice. Axis does not expose the HTTP session in any way, but some sort of manipulation of Axis could have solved it (e.g. sublassing the Axis Service servlet or with AOP) This is obvious a horrible dependency, but luckily we could get access to the source code and easily patch it.
My coworker coined the phrase I had been thinking of for libraries and components that pulls data from unknown sources as pulling from a black hat.
Another coworker who had been working a lot with this library said this dependency had caused him a lot of trouble while writing JUnit tests. I happen to be Testinfected, and believe libraries/componentes that make Unit tests hard to write should be considered candidates for refactoring and even replacement.

If you encounter Black Hat dependant libraries or components that makes Unit tests hard to write, consider your options:

Do you have the the source code?

Can you change it?

Is it easy to replace? In the long term it can harm your project.
Do you need it? Can you develop a replacement with the same or less effort as the hazzle struggling with it?