Tag Archives: IT

How (and why) to move performance testing earlier in the development cycle [WOPR22]

I recently attended the WOPR22 conference in Malmo. This focussed on discussions around how to move performance testing earlier in the process. This is a big subject and there is clearly no magic bullet solution, but I thought I’d share some of the key takeaways from the discussions.

Performance testing needs to be thought of as more than just load testing

A traditional approach of “finish development, go through a load testing process and approve/reject for go-live” really doesn’t work in a modern development environment. The feedback loop is just too slow.

We have to be providing feedback earlier. Load testing though is not ideal for this, there are a lot of drawbacks – scripts have to be maintained, environments available, datasets managed, load models understood. These are surmountable but quicker feedback can be provided by simpler processes that can be added to a standard CI solution or executed manually but regularly.

Examples are things such as

  • waterfall chart analysis of pages being returned
  • unit test timings
  • parallelisation of unit tests to get some idea of concurrency impacts
  • using perf monitors/APM on CI environments

These will not find true problems that only occur under load but they will give you some early indications that problems may be there.

Performance Testers need tighter integration with the development team

The performance testing team cannot be too distant from the development team – for practical and political reasons. There was a lot of discussion about whether a performance team works best as a distinct entity or as individuals integrated across the teams. There are pros and cons for each argument.

What is clear though is that when issues are identified there must be co-operation between performance testers and developers to share their knowledge to resolve the problem. Performance testers should not be people who just identify problems – they must be people who are part of the team that solves the problem.

Mais Tawfik has the policy of physically pairing the performance tester who has identified the problem with the developer who is working on the fix until a resolution is found.

Performance Testers still need space for analysis

One of the downsides to pushing performance testing earlier was that it often results in an additional demand for testing without provision of appropriate space for analysis. Performance testing is an area where analysis of the data is important – it is not based on black and white results.

It is often overlooked that data is not information. Human intelligence is required to convert data into information. An important role of the performance tester in improving any process is to ensure that there is not an acceptance of data over information because data can be provided more regularly. We must ensure that there is sufficient quality, not just quantity of performance testing during the development process.

Environmental advances can make this process easier

Cloud and other virtualised environments as well as automation tools for creating environments (e.g. Chef, Puppet, CloudFormation) have been game changers for earlier and more regular performance teasing. Environments can be reliably created on demand. To move testing earlier we must take advantage of these technologies.

Use automation to simplify the process

Automate the capture of metrics during the test to speed up the entire process. Using APM tooling helps in this respect. Automating this reduces the overhead associated with the process of running a test and analysing results.

Attendees of WOPR22 in Malmö, Sweden.

Attendees of WOPR22 in Malmö, Sweden.

Based on discussion with all WOPT22 attendees:

Fredrik Fristedt, Andy Hohenner, Paul Holland, Martin Hynie, Emil Johansson, Maria Kedemo, John Meza, Eric Proegler, Bob Sklar, Paul Stapleton, Neil Taitt, and Mais Tawfik Ashkar.


Leave a comment

Filed under Opinions

6 Rules for Good Specifications in Software Development

In recent months we have overhauled the Intechnica development process to use a lot of the BDD – Behaviour Driven Development –  and “Specification by Example” (http://gojko.net/) processes and tools for specification definition. As such we have defined the 6 rules of good specifications:

1. Communication between client, BA, QA and developer:
The more all of these people talk, and the earlier they are involved in specification stage, the better.

2. Don’t define user experience and business functionality in one place.
It leads to repetition and confusion. It also tends to lead to underlying systems designed around the UI rather than the business domain.

3. Illustrate all business logic with examples
…not with wordy descriptions.

4. Illustrate all UI with annotated pictures
…or ideally full HTML prototypes.

5. Specifications must be testable to form objective acceptance tests.
Tests should be automated and built into your continuous integration process.

6. Specifications should be living documents
…and evolve into the living documentation of the system.

View some of Andy’s presentations about software development on Intechnica’s SlideShare


Filed under Opinions

First thoughts on Windows 8 – the Developer standpoint

I’ve been to a few demos recently of the full Windows 8 experience so thought I’d put a few thoughts down about it.

For those who have not seen it, Windows 8 really is “revolution not evolution”. It throws away many of the concepts we have been familiar with since Windows 95 (start menu, task bar…) and even some of the core concepts of Windows such as … well… Windows. It is the first OS designed to work across desktops and tablets.

The core of the new system is the Metro interface which genuinely doesn’t have windows. All applications are run full screen for what (I have heard called many times during these demos) is a “fully immersive experience”.

Windows 8 will seamlessly integrate between desktop, tablet and mobile.

The standard means of navigating around and between programs have been largely removed. There are no more close and minify buttons or menu bars. Instead addition options for programs are found by swiping the appropriate part of the screen. The right hand side is for the “charm” screen which is a set of common options that apply to all programs (notably search and share), while top and bottom can be used for application options. Programs are closed by dragging them off the bottom, and minimised by dragging across and bringing another program on.

This leads onto another key change: when a program is inactive its processing is suspended. To a certain extent this can be developed around but by default Metro apps do not allow multi-tasking.

The Metro screen displays all apps as tiles; tiles and other notifications can be fed information from within your application or pushed from remote services via a cloud based distribution service. This leads to a dynamic desktop and gives developers the ability to entice users back into their application.

There are probably a lot of people thinking that this will mean the end of traditional windows based applications but worry not, there is also desktop mode. In desktop mode traditional Windows programs will run in the same format as they always did. Desktop apps can be launched via shortcuts from the Metro tile screen or from a separate menu within desktop mode.

As always with Microsoft products the support and facilities given to developers for the new platform is extensive and supports all the systems needed for creating innovative apps but that is a subject for a blog post of its own.


So, what do I think of all this? Metro is clearly designed to be a tablet interface; indeed Microsoft have said that it is designed for touch first and everything else follows out of it. As a tablet interface it is very innovative. The sharing ability between programs is excellent and a big step forward over anything that has been seen before. The built in cloud support for multiple devices is also a great feature, allowing synchronisation across multiple devices straight out of the box.

I think it works less well in a desktop environment; I have been trying to think about what applications I would use in Metro mode. Twitter, email, IM, and web browser immediately spring to mind. However I spend most of my time in Visual Studio, SQL Server, Office or other such tools and usually have the other tools mentioned above open at the same time, usually on a separate monitor so I can watch over them while carrying on working. Ultimately, I have paid for a large screen and I don’t want a fully immersive experience; I want to control my level of immersion by setting windows to the size I want.

The other issue I have is that, despite how many times I hear or read about Microsoft people saying how beautiful the Metro interface is, I am not wowed by it. I have never loved the look and feel of Windows phone and Metro is essentially and evolution of that theme. The tiles are great in their ability to stream real-time information outside of the application, but I find the look too busy and I’m really not keen on the way things flip in and out. Basically, I look at the Metro interface, I look at my iPad, and I’m not wanting to throw out the iPad.

The Windows 8 logo

Building a standard system across all platforms is a bold move and only time will tell how well it will work out. It is fair to say that Windows 8 is a pretty good stab at it.

What I find interesting is that to me this is much more of a consumer than a business solution. I feel that many of the business users who, like me, spend all day in tools like Visual Studio or Office may share my concerns specified above. Consumer users though will probably love the concepts behind the Metro interface. In this market a tablet is a serious alternative to a laptop to achieve their aims (web, email, twitter, IM, Skype etc).

This raises an interesting question about whether the historical vision of Windows as a shared system between business and consumers is actually flawed and Microsoft should be looking at selling Metro only (with no desktop) as a consumer option and a full Windows 8 system with desktop support for business users. The consumer version could even be a free offering (like iOS) that is funded by the profits on app sales.

I started by saying that Windows 8 was “revolution not evolution” and am going to end by saying that it may be revolution beyond just UX, and go to the heart of Windows as a product.

Leave a comment

Filed under Opinions