Wednesday, May 20, 2009

Sysinternals (Process Explorer) saved my a**

After deploying our .NET system in production functionally speaking everything was OK. However, after couple of days we saw our processes consuming more than 80% of the CPU of some pretty powerful servers - and we were not computing any nuclear or medical killer stuff. Something was wrong - but what?

At this moment I recalled using free Sysinternals tool from Microsoft to see what's deep inside my operating system (in runtime, of course).

What really happened?
I started Process Explorer belonging to the Sysinternals suite, found my process and looked at it's runtime properties:


What I saw was the increasing number of TCP/IP connections (because used connections were not closed/returned to the pool properly). After some time spent on monitoring the process I noticed that this pattern is stable and each request caused the number of connections to database grow - until the process reached max and OS couldn't handle this process any more. Of course I fixed this issue wit one or two lines of code because it was a "little" problem (as usual) that appeared to have a big impact on the operating software.

I think wouldn't be able to discover this issue so quickly without using Sysinternals. Why? Because my colleagues were trying to investigate and fix this issue for some time before me. They didn't know Process Explorer tool :)

Monday, May 18, 2009

Direct communication as a half-dead project saver: When you start in an over-waterfall company

Picture courtesy of clagnut@flickr
At the end of year 2008 our team was given a project that was critical from the $$ point of view. We had to deliver the project by the end of March 2009 in order to avoid huge fees from our customer. The problem was that the project we got was being developed by different team in different location and the quality of the existing solution was at least questionable. It turned out that the whole old team left the company and our small commando squad had to save the day. I just have to mention that the company we all worked for was a pure-waterfall monster.

You know, agile books are fun to read, but how do you start it if you happen to be a developer in a pure waterfall company on an impossible project? I went through it and going to show what worked well for us. I will tell how we were able to get an impossible project apply the agile principles and survive.

A bit of a history
The company I used to work for sometime didn't know/use Agile methods. Everything there was pure waterfall i.e. there was a marketing that defined what it wanted (previously negotiated the contract i.e. feature list with the customer), then it passed the specification docs to product definition team that was in charge of designing the solution, finding ALL possible problems and answering ALL possible questions. Then after couple of months the technical specifications were passed to the development team that... started everything all over again. I mean that the development team usually found documents received from product definition team useless - it's not my opinion, rather my conclusion after talks with many friend-developers. Some docs like protocol specifications or conversion specs were REALLY useful but they were vast minority.

Anyway, when developers started working on the project it usually took just few classes to find some blocking issues that had not been predicted/designed in the docs. "How it's possible?" "They were supposed to find ALL possible problems and answer ALL possible questions". Well, they were supposed to but it was IMPOSSIBLE to do so. Even perfectly prepared and thoroughly checked documentation will not replace direct communication channels.

The project I'm going to describe was quite different - there was no documentation, no specification but more so there was not a single person understanding the problem. This was a real shocker for the company used to work in an "ordered" and "predictable" way.

The Project
In short we had to develop an unknown communication protocol to connect the two unknown existing systems. And the time we were given was pretty much estimated as we already knew all the guts and gears in this system. Ah - I forgot about some pretty "unimportant" detail: the test database for the test system was located around 1000 km from our desks and the VPN connection was not yet established. So, for couple of weeks we were just having a pretty "black box" environment - it was not so bad though because we had some time to learn how to compile given sources (still, this was the time when we were supposed to be developing software).

What did we have?
We had the documentation for the protocol we had to implement with examples (requests and responses). The documentation was not very useful, though because it was designed for people who knew this protocol before. This documentation was not explaining many concepts - it was assuming the reader's knowledge on quite high level - unfortunately we were GREEN.
We also had some stored procedures (in our unreachable database) that were supposed to contain business logic and they were supposed to work already - we were told that they require just small changes. Basically we had to develop a part that had to receive the requests, parse and process it and invoke already existing stored procedure with transformed parameters – we knew that we would also modify or create completely new stored procedures from scratch.

When we eventually got to know what to do we knew that we had to implement hotel availability (3 different request/responses) and booking (couple of quite complex transaction flows) messages. The availability part was supposed to be 100% ready (this way we could focus only on booking) - we just had to test it and confirm (probably do some small fixes).

To summarize: we had some germ of the C# code, fully functional stored procedures in T-SQL (MS SQL server) that required some significant rework and proprietary ASCII format specification. I also have to mention that neither of the team members was C# or T-SQL expert (I put it as delicate as possible).

Finally @Work
After some time of investigation we knew how the frontend and backend work but we still couldn't test the system because we had no database. When it was ready we established a test environment and we finally was able to integrate the whole system. We were able to start this machinery and inject some requests. We were even receiving some responses - but at this moment we had completely no idea whether these responses were correct or not.

Here is the moment we started communicating directly with our customer who started testing delivered system in test environment and giving us feedback. If you ask why we didn’t communicate with them earlier I would answer that we had nothing to show them as the test environment was unreachable for developers. If we couldn’t show them anything we weren’t able to get any feedback. That sucked…

I will leave the details of our communication process with customer for the next post.

Lessons learned so far
We took over the project in a rush and were just given a source codes from the previous team – which wasn’t the first class anyway (I mean the code). There was no unit tests, no continuous integration environment, no single product backlog but we started creating those artefacts. And you know what – this was the best decision we could make - we took as much as we could from the agile world in this project.

The main problem that was probably the root cause of all other issues was lack of communication. Before starting contacting directly with customer (who became our product owner, in fact) this was the worst part of the project because we not only didn’t know what to do but also we had no one who could verify our assumptions. Lack of communication was very visible and the deadlines were getting closer. We were really thinking that we will be late with such pace and information flow (or rather lack of it). We knew that with such level of uncertainty agile practices - especially open and effective communication - could help us saving our jobs.

In the next part I will explain how we solved our communication problems, how agile practices helped keep the project on track and keep the number of bugs close to zero as well as how the whole project finished. Stay tuned!

Feedback
Do you recall such situation in your own project(s)? What was the solution that worked for your team? I'm really curious your opinions and thoughts.

Originally published on AgileSoftwareDevelopment.com

Friday, May 15, 2009

Niezły kawałek muzyki

This post will be in Polish but don't worry it won't be about Java or technology at all. Anyway, if you want to listen to some really good music (one of the artist - Piotr Czerski - is my friend from university and some really wasting parties ;) visit this website: http://towary.art.pl.

Mam nadzieję, że Groszek się nie obrazi jeśli skopiuję post z jego bloga - nic lepszego i tak nie wymyślę na temat nowiuteńkiej płyty grupy "Towary Zastępcze" pt. "Dolne Miasto" bo o tym będzie ten wpis. Przesłuchałem tę płytę już dzisiaj dwa razy i jestem bardzo kontent.

A oto co do powiedzenia ma na temat płyty Groszek (ze wszystkim się w stu procentach zgadzam):

Na płytę składa się dwanaście zaśpiewanych wierszy, opowiadających strzępy tragicznej historii dziewczyny z Grand Hotelu. Szczególną uwagę zwraca strona internetowa towary.art.pl promująca album, na której możemy wcielić się w rolę detektywa prowadzącego śledztwo. Oglądając kolejne fragmenty video i podążając właściwym tropem, z pewnością uda nam się zdobyć odpowiedzi na trzy pytania, co pozwoli ściągnąć album w formacie mp3 za darmo z sieci. Wytrwali mogą również odnaleźć prawdziwego mordercę i dotrzeć do kolejnych bonusów.

Album dostępny jest również w bardziej namacalnej postaci. W atrakcyjnej cenie można nabyć akta sprawy, w których jednym z dowodów jest płyta. Doskonały pomysł na nietuzinkowy prezent i wsparcie twórczości artystów. Gorąco polecam!


Ja też polecam - kawał dobrej roboty i muzyki!

Java Concurrency Bugs/Programmer's Errors

Alex Miller has been posting very interesting series on Java concurrency bugs (or pitfalls programmers could fall into). Especially interesting is the last post: Inconsistent synchronization. This post is most probably inspired by the problems Alex discovered in Hibernate statistics and I am pretty proud I was able to find possible concurrency issues reading the code. Anyway, I was not able to fully and correctly solve those issues.

The conclusion for me is that I thought about "magic" solution instead of the correct one i.e. I suggested that declaring a integer field as volatile would solve the issue. In fact it would but the "correct" solution to this problem is using AtomicInteger from java.util.concurrent.atomic package.

Another conclusion is to remember to keep bombarding your Java code with FindBugs - it can find many many Java issues in your code (not only related to concurrency).

The rest of the Alex's series (so far) is here:
Java Concurrency Bugs #4: ConcurrentModificationException
Java Concurrency Bugs #3 - atomic + atomic != atomic
Java Concurrency Bugs #2 - what to synchronize on
Java Concurrency Bugs #1 - mutable statics

Enjoy you Java Concurrency Issues ;)

Monday, May 04, 2009

Fun soft: regex checker

I'm not a regular expressions expert but I use them often - as probably most of software developers. Before using them I want to be sure my regexes are written correctly and they will serve their purposes. I wrote very small command-line app for doing the check for me :) As a first input user provides the regex to check, then the user can type all texts to check against previously given regex until she types 'q' or 'exit' (for which program exits). Enjoy!

package com.bielu;

import java.io.Console;

public class RegexTester {
public static void main(String[] args) {
Console in = System.console();
if (in == null) {
System.err.println("Cannot obtain system console - exiting application");
return;
}

System.out.print("Enter regex to test: ");
String regex = in.readLine();

while (true) {
System.out.printf("Type value to match with given regex '%s': ", regex);
String string = in.readLine();

if (string.equalsIgnoreCase("q") || string.equalsIgnoreCase("exit")) {
break;
}

if (string.matches(regex)) {
System.out.printf("'%s' matches '%s' regex\n", string, regex);
} else {
System.out.printf("'%s' does NOT match '%s' regex\n", string, regex);
}
}
}
}