Tuesday, September 30, 2008

When to use synchronized or Lock / Condition objects?

In this post I will present when you should use synchronized keyword in your Java code and how you can easily exchange it with Java 5 concurrency features.

Let's create Bank class that has an array of accounts and can transfer money from one account to the other. Of course, it can be done in multiple concurrent threads and thus wee need synchronization mechanism.

Consider following implementation of the Bank class using locks and conditions:

package locks;

import java.util.Arrays;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

class Bank {

Lock bankLock;
Condition sufficientFunds;

double[] accounts;

public Bank(int accounts, double initialBalance) {
this.accounts = new double[accounts];
Arrays.fill(this.accounts, initialBalance);
bankLock = new ReentrantLock();
sufficientFunds = bankLock.newCondition();
}

public void transfer(int to, int from, double amount) {
bankLock.lock();
try {
System.out.printf(
"Checking funds [%.2f] on account [%d]: [%.2f]\n",
amount, from, accounts[from]);

int count = 0;
while (accounts[from] < amount) {
sufficientFunds.await(5, TimeUnit.SECONDS);
count++;
if (count > 3) {
System.out.println("--- could not transfer money ---");
return;
}
}

System.out.printf(
"Transferring [%.2f] from [%d] to [%d]\n",
amount, from, to);

accounts[from] -= amount;
accounts[to] += amount;

sufficientFunds.signalAll();

} catch (InterruptedException e) {
System.out.println("Gave up the lock...");
Thread.currentThread().interrupt();

} finally {
bankLock.unlock();
}
}

@Override
public String toString() {
bankLock.lock();
try {
double sum = 0;
for (double account : accounts) {
sum += account;
}
return "Bank's total balance is [" + sum + "]";
} finally {
bankLock.unlock();
}
}

public int size() {
return accounts.length;
}
}

Some explanation: before transferring the money I lock the bank object and wait for the sufficientFunds condition in the while loop with appropriate condition statement. If the condition is signaled or timeout passes but the condition is not met more than three times this program gives up transferring the funds and prints "--- could not transfer money ---" message.

Here is the source code to run the example:

package locks;

public class LocksAndConditionsTest {

private static final int ACCOUNTS_COUNT = 10;
private static final double INITIAL_BALANCE = 1000;
public static final long DELAY = 500;

public static void main(String[] args) {
Bank bank = new Bank(ACCOUNTS_COUNT, INITIAL_BALANCE);

for (int i = 0; i < ACCOUNTS_COUNT; ++i) {
Thread t = new Thread(new TransferTask(bank, INITIAL_BALANCE));
t.start();
}
}

static class TransferTask implements Runnable {
Bank bank;
double maxAmount;

public TransferTask(Bank b, double maxAmount) {
this.bank = b;
this.maxAmount = maxAmount;
}

public void run() {
try {
while (true) {
int to = (int) (bank.size() * Math.random());
int from = (int) (bank.size() * Math.random());

if (from == to) {
if (to == 0) {
from = to + 1;
} else {
from = to - 1;
}
}

bank.transfer(to, from, maxAmount * Math.random());
Thread.sleep(DELAY);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
}

Here is the source code for the Bank using synchronized keyword and old wait / notify mechanism:

package locks;

class BankSynchronized extends Bank {

public BankSynchronized(int accounts, double initialBalance) {
super(accounts, initialBalance);
}

@Override
public synchronized void transfer(int to, int from, double amount) {
try {
System.out.printf(
"Checking funds [%.2f] on account [%d]: [%.2f]\n",
amount, from, accounts[from]);

int count = 0;
while (accounts[from] < amount) {
wait(5000);
count++;
if (count > 3) {
System.out.println("--- could not transfer money ---");
return;
}
}

System.out.printf(
"Transferring [%.2f] from [%d] to [%d]\n",
amount, from, to);

accounts[from] -= amount;
accounts[to] += amount;

notifyAll();

} catch (InterruptedException e) {
System.out.println("Gave up the lock...");
Thread.currentThread().interrupt();
}
}

@Override
public synchronized String toString() {
double sum = 0;
for (double account : accounts) {
sum += account;
}
return "Bank's total balance is [" + sum + "]";
}
}


To test this implementation just change
Bank bank = new Bank(ACCOUNTS_COUNT, INITIAL_BALANCE);
to
Bank bank = new BankSynchronized(ACCOUNTS_COUNT, INITIAL_BALANCE);
in the LocksAndConditionsTest class and start the main class again.

The behavior of these two implementations should be the same (well, in fact it should be similar according to the Math.random() usage).

Implicit locks and conditions (synchronized keyword) have some limitations:
  • You cannot interrupt a thread that is trying to acquire the lock

  • Having a single condition per lock can be inefficient


And when you take a look at the JDK documentation you can find this:
Condition factors out the Object monitor methods (wait, notify and notifyAll) into distinct objects to give the effect of having multiple wait-sets per object, by combining them with the use of arbitrary Lock implementations. Where a Lock replaces the use of synchronized methods and statements, a Condition replaces the use of the Object monitor methods.


When to use Lock, Condition objects or synchronized methods:
  1. It is best to use neither Lock / Condition nor the synchronized keyword. In many situations, you can use one of the mechanisms of the java.util.concurrent package that do all the locking for you.

  2. If the synchronized keyword work for your situation, by all means, use it. You write less code and have less room for error.

  3. Use Lock / Condition if you specifically need the additional power that these constructs give you


As you can see the main conclusion is that you should avoid both mechanisms like fire. You should use java.util.concurrent features. More on this in the next article. Stay tuned.

References: JDK 6 API and Core Java 2, Volume II--Advanced Features (7th edition)

Java concurrency is easy

I'm going to start the series of posts about Java concurrency in practice. More accurately I want to show everybody that with the new Java 5 (and 6) features concurrency in Java is very easy.

It is very exciting for me as I graduated from Distributed Computing specialty at my university and I love everything that has more than one thread in its subject. Concurrency issues are very interesting but many people ALWAYS say that it's difficult subject, blah blah blah. It's like with the maths at school - it's the best and most interesting subject but everybody who doesn't even try to understand it says it's difficult. It's really harmful because it discourage other people from even taking a look at this - "if someone says it's difficult it must be difficult". Bullshit!

Java concurrency is EASY!

Stay tuned for the first article this week.

Friday, September 26, 2008

Solution to Struts2 upload file error: the request was rejected because its size (XXX) exceeds the configured maximum (YYY)

The error message from the title is generated by Commons FileUpload and caught exception in class org.apache.struts2.dispatcher.multipart.MultiPartRequest is not translated to any custom message. It causes the problem with i18n because in polish or french application I would not appreciate english message. There are at least three solutions to this problem (all of them are pain in the ass): one is to reimplement org.apache.struts2.dispatcher.multipart.MultiPartRequest class; second one is to reimplement org.apache.struts2.interceptor.FileUploadInterceptor class. I found a third solution which is the least painful but as well ugly.

All you have to do is to implement validate() method in your Struts2 action class like this:
 
@Override
public void validate() {
Collection<?> tmp = getActionErrors();
Collection<String> errors = new ArrayList<String>();
for (Object o : tmp) {
if (o.toString().contains(
"the request was rejected because its size")) {

errors.add(getText("error.exceededUploadSize"));
} else {
errors.add(o.toString());
}
}

setActionErrors(errors);
}


Enjoy!

Wednesday, September 24, 2008

Seven Principles of Lean Software Development - Deliver Fast

Picture courtesy of johnthescone@flickr
Work in small batches
I've always thought that delivering small pieces of software is easier than delivering the BIG BANG product at once. Small software package is easier to manage than the big one, you can expect smaller amount of defects there, it is much easier to integrate it with existing software. Isn't that obvious?

On the other hand working on the new version for, let's say, six months without regular integration in the production environment (if integrated at all) will bring you much more problems. Defects will accumulate and just before the release someone will discover them. No surprisingly developers would have to work overtime to fix them.

More so, let's say that you have to spend always additional 10% of time on integration and bug fixing of delivered package (working either on small or large batches). If you deliver products each 4 weeks (20 days) fixing and integrating the package will take you two days (for the customer it's almost transparent - nobody will even notice two-day delay). If you deliver new versions each 6 months (130 days) fixing bugs and integrating the product will take almost two weeks (if defects will not accumulate) - your customer could go mad about it.

That's why you should reduce projects size, shorten release cycles, stabilize work environment (listen to what your velocity tells you), repeat what's good and eradicate practices that creates obstacles.

Limit work to capacity
According to Mary and Tom Poppendieck you should limit tasks queue to minimum (one or two iterations ahead is enough). You should not be afraid of removing items from the queue - they even claim that if you remove something important it will not do any harm because your customer will remind you about it pretty soon! Keep only important, valuable and urgent items there i.e. features that bring biggest business value to your customer.

Give your team some time to stabilize velocity, to see how much you can deliver each iteration. This is the only reasonable and REAL value. You cannot assume that your team will deliver twice as much next month - you know your capacity. Expect your team to pull items from the queue based on measured velocity and to completely finish the work before they move to the next ones.

Moreover, you should reject any work until you have an empty slot in your queue. It is really great tip because it doesn't make sense to add items to the backlog if you know you won't have time to implement them, unless you want to add something more important than items that are already there.

Focus on cycle time, not utilization
Many software organizations want their engineers to be utilized at 100%. It's not a good idea, though. Take your computer for example or server machine. What do you think if its CPU is utilized at 100%? Yes! You try to figure out what process you can kill to reduce the utilization.

Full utilization creates traffic jams which in turn reduces overall utilization. Queuing theory applies here i.e. you should put in your queue small tasks that cannot clog the process for a long time. You should reduce cycle time and have fewer things to process in your queue.

Also, instead of worrying about the resource utilization you should start measuring time-to-market and customer response time. Smaller the batches and shorter the cycle time the smaller those values will be.

Wrap up
I hope pieces of advice given above will make it easy to understand how to put "Deliver Fast" principle in practice. If you need more detailed description with more examples and more sophisticated explanation you should definitely go to "Implementing Lean Software Development - from Concept to Cash" book.

Originally published on AgileSoftwareDevelopment.com

Thursday, September 18, 2008

Seven Principles of Lean Software Development - Respect People

Picture courtesy of Elvire.R.@flickr
Software organizations have surprisingly more problems managing people than managing soulless machines. People are also surprised when they ask me what my job is all about and I tell them that I work with people most of the time. "But you are a software engineer - isn't your job about working with computers?". "You're partially right because I work in a team and program a computer using programming language to do what you want to do is much easier than >>program<< my colleagues to work with me to reach our team's goals" - I tell them.

Working in a team means not only solving technical problems, designing stuff and make decisions together. One of the most important factor that makes teams successful is Respect.

In this post I will try to explain "Respect People" principle from "Implementing Lean Software Development - from Concept to Cash" book.

Group or Team?

Do you work in a group or a team? If you can see your work environment objectively you will know whether your colleagues are eager to work and solve technical problems in your project or rather they are told what to do by the manager. If you are told what to do i.e. your team lead decides what you should work on next (or even tell you how to work) you work in a group.
On the other hand if you meet often and regularly with your team and everybody is eager to work and offers help to each other it means that you work in a team.

Teams Thrive on Pride, Commitment, Trust and Applause
What is the difference between a group and a team? As the title of this paragraph says, teams thrive on pride. It means that teams are proud of what they do and how they do.
Teams thrive on commitment i.e. team members love to be committed to some work - they are eager to work (commit) and do it with pleasure.
Teams thrive on trust i.e. team members trust each other and know that if they have problems they can count on colleagues - they will never be left alone with their problems.
Last but no least, teams thrive on applause i.e. they want to know that they are doing useful software and that they bring value to the organization they work for.
Teams have commons goal(s) and they pursue it together. Team members respect each other and create great places to work for everybody.

Group is just a couple of individuals
What about a group? Well, a group means that there are some engineers who basically work for the same project and they do not mutually commit to accomplish project goals. They work separately and from time to time merge the effects of their work.
BTW- example: I noticed that some of the new engineers in my company are just left alone after the first couple of hours when they arrive. Senior guys who should show them everything just give them hundreds of pages of documentation hoping they will find something useful in there and wait until they say they are ready to get some assignments. This is not a team!

If you feel you work in a group and would like to work in a team there is a remedy for you or your managers.

Steps to achieve the goal

Here are three steps authors of this great book recommend implementing in every lean organization.

Train team leaders/supervisors
Give team leaders the training, the guidance and some free space to implement lean thinking in their environment. Do not impose anything - they know better how to implement it in the areas of their responsibility.

Move responsibility and decision making to the lowest possible level
Teams should deliver value under the guidance but they should not be driven by leaders or managers. Let your people think and decide on their own - they know better how to implement difficult algorithms and apply stat-of-the-art software frameworks. Leader's role is very important here - they should be team's enablers and facilitators.

Foster pride in workmanship
Remove impeding practices like individual rewards (foster team rewards, instead), careless workspaces and work practices (no testing, no coding style, merging big pieces of code, etc.), robot-like jobs, etc. You should encourage passionate involvement of your team members to what they do. Let your people decide what and how to do in order to meet goals, don't depreciate and disregard people's estimates (yes, it is sometimes so difficult to implement it) - and remember that they are only estimates. This will have a more sustained positive impact by far than individual incentives and bonuses.

That's it! I know it's a bit shallow and I may missed some points but it is anyway a good starting point for you to apply "Respect People" principle. I hope I explained it in a way you are able to apply it straight away. If you would like to know more and to read some more sophisticated examples go to "Implementing Lean Software Development - from Concept to Cash" book.

PS. Remember to respect not only people at work but all the people - this will definitely pay off (even if it will not pay off you should respect everyone).

Originally published on AgileSoftwareDevelopment.com

Friday, September 12, 2008

Can unit testing be a waste?

Picture courtesy of Kecko@flickr
It is sometimes hard to start implementing new feature (or fix a bug) by writing an unit test even for a very agile and experienced developer, that's for sure. But it pays off and it usually turns out to be the easiest and the most efficient way. Besides this, developers have to decide what to test. "You should test everything!" some of you may say, but what does it mean?

Should I test all my JavaBeans (as Java is the language I mostly use I will give examples in this technology)? Should I test all my Data Access Objects? Should I test all toString(), equals(), hashCode(), etc. methods? Or maybe I should test only stuff where I "integrate" all those "low-level" components/classes?

I'll try to answer the question of what level of abstraction should be tested by your unit tests and why some unit tests may be considered a waste in this post.

Testing everything (literally) is a waste
You should start developing new features from unit tests. More so, you should start by writing tests from the highest possible level of abstraction. I mean that if you want to test adding new users to the database feature you should start from the service class that takes input parameters, creates user object and then stores it using some kind of DAO object into the database. You should not start from testing and implementing User class or UserDao class.

Why? Well, if you start thinking of objects you will need for implementing your feature you may invent some overkill architecture, many unnecessary classes that will eventually be unused. With this approach you will start from implementing User class, then UserDao class and eventually you will implement the service class that will use these both classes and expose some public methods to the outer world.
With this approach, if you are still agile and start from writing an unit test, you should end up with at least three unit tests (one for each class). And assuming that User class is stupid JavaBean with only getters and setters you will have to write unit test for those methods, right?

This is a waste (especially unit testing getters and setters) because you could have only one unit test class with less code and have 100% code coverage.
Agile methodologies focus on eliminating waste - that's why you should not test everything if you want to be really agile.

The only way is to start from the highest possible level of abstraction and then adding necessary classes just-in-time. And the example follows...

What should you test
I will not give you an example of adding users to the database but instead an example of Struts2 action that retrieves announcements from the database and creates RssChannel object (that is processed by the View layer later on and returned as an application/rss+xml content type that is readable by the RSS readers). Take a look at the attached RssAction.java file. This action uses other classes like:

  • AnnouncementDao

  • RssChannel

  • Announcement

  • ...


Now take a look at the test for this action: RssActionTest.java. At the first sight this unit test tests only com.bielu.annoboard.action.rss.RssAction but in reality I also tests all classes mentioned above.

The effect? Cobertura tells me that:

  • RssAction is covered at 100%

  • RssChannelHelper is covered at 92%

  • RssChannel is covered at 95%

  • AnnouncementDaoImpl (which is an implementation of the AnnouncementDao) is covered at 100%

  • Announcement is covered at 100%


Voila! No waste and everything (or almost everything) is tested!

Use your common sense
As always there are some pitfalls lurking - my previous opinions are not always true. For sure you should NOT test getters and setters in your JavaBeans. But what if you implement for example toString(), equals() or compareTo() method? Should you test them or not?

I generally test classes from the highest possible abstraction level but I sometimes also test lower-level classes. Here are some pieces of advice you should consider following:

  • it is a very good idea to test toString() method

  • it is an absolute MUST to test (to death) equals() and hashCode() methods and contract relations between them (refer to "Effective Java" book) - if you use Sets, Maps and collections in general this is a critical part - I even remember the very small bug there in one of my previous projects and two seasoned Java developers looking for the problem for couple of hours - this is my coding horror

  • it is a good idea to also test compareTo() and contract relations to equals() method

  • check with your code coverage tool what is still not tested after the high-level tests

  • consider removing the code that is not tested and see whether it was used at all (see this post)

  • if it is necessary but not tested then test it from the lower level


Use these pieces of advice as well as your commons sense wisely and remember to eliminate waste every time you can (yes! writing unnecessary unit tests is a waste).

I'm really curious about your opinions about writing unnecessary unit tests. Do you think writing unnecessary tests can be even dangerous? Do you consider unnecessary and overlapping unit tests a waste? Share your opinions here.

Originally published on AgileSoftwareDevelopment.com

Wednesday, September 03, 2008

Don't bring me solutions, bring me problems

Picture courtesy of Liz Henry@flickr

Title of this post is a paraphrase of great quotation, namely "Don't bring me problems, bring me solutions" which is or rather should be sometimes used by project leaders and product owners (also referred as managers later in this post). However bringing problems to Scrum Master is a very good idea - some problems are out of your control, especially those managerial ones (but this is a subject for separate post).

We, engineers, are taught to solve problems and we do it right. We are creative and intelligent people, hungry for non-trivial puzzles to solve. We should receive problem as an input and give solution as an output - that's it (of course we need a lot of communication in between, clarifications, details, etc.).

Why managers/leaders/product owners sometimes treat developers as machines to write the code? Why they don't trust developers and don't believe they really have knowledge and skills to solve many business and technical problems and turn them into working software? Why don't they accept the fact that when development team encounters any problem they will try to solve it and if they fail they will let them know.

Of course, it's not fair to say that every manager and leader has such mental problems I described above (I was lucky for most of my professional life) but let's focus on guys that really think they know everything and thus kill team's creativity.


Where all think alike, no one thinks very much -- Walter Lippman
Imagine such situation: as a developer you are asked by your manager to implement some new features. You discuss with her the details, input/output, UI, whatever... When you are done and you know roughly what to do she starts describing you how she would like to have it implemented. She talks about technology, about probable problems you may encounter, about technical decisions you have to take. Basically she gives you one of the solutions just like that.

What would you think after such conversation? Personally, I'm getting furious inside because I've just been treated like brainless, thoughtless, whaterverless coding "professional". I always wonder why they need me if they don't even want me to think. I feel useless after such conversation.

You just got the problem to solve but a second after you also got a solution (to be more precise PROPOSAL of the solution). If I think like my manager it seems I don't think too much - if I try to modify her solution (which is likely to happen because I have other ideas and I know technical constraints better) I may have problems during yearly evaluation... You know, people often treat their ideas like their children - would you dear to say your manager's child is stupid? I don't think so.

Suggesting solutions kills creativity
It's true - suggesting solutions drives you to them, even if you would invent something more creative and smarter. If you hear about the possible solution to the given problem you are somewhat contaminated by it and it's really hard to get out-of-the-box. It's really hard to think about all technical consequences of such solutions because when you hear them you may think "Hmmm, that could really work - it may be a good idea".

Suggesting technical solutions to engineers is sometimes dangerous. Manager's solutions can look good at first but as manager is not "deep inside" there and doesn't know all technical constraints they can lead to serious project's delays. I know about many such managerial "solutions" that were very crappy; "solutions" by which teams were loosing precious time. These "solutions" were not stupid or bad - not at all. But those solutions didn't take into account many technical aspects managers couldn't know - only developers knew about them (see tacit knowledge).

Don't bring me solutions, bring me problems
Most of the times development team members know better how to solve problem given by product owner or manager. If you are product owner you should focus on the feature from the user's perspective - development team will deliver you technical solution. Even if you have some ideas you should probably wait with telling about them until developers ask you if you have one. Giving or suggesting technical solutions to engineers is example of micromanagement IMHO and should not be emulated.

Product owners/managers - bring developers problems, not solutions. If you bring engineers solutions it means that you don't need them. And it's not nice to be a developer then. If I feel useless I will not deliver good quality stuff and I will probably start looking for another - more challenging - job. Engineers need responsibility, they need to be owners at least of their solutions.

Next time when someone from the management or product side will give you problem together with solution just tell them that if they are so smart they should start developing it themselves. They have everything they have to know. If you think it's too drastic solution just tell them "Don't bring me solutions, bring me problems"


What do you think? Should managers give us solution or only the problems? What is the value of such managerial "solutions"?

Originally published on AgileSoftwareDevelopment.com