Bugs that are hard to catch

I ran into a bug today in code that is one of those impossible to catch.

fieldValue = format(trnx_count, “0000″)

trnx_count is a string variable that consists of hexadecimal digits. If you have trnx_count = “58″, then field value will equal to “0058″.

But………….

When you have trnx_count = “01D7″, which is a valid hexadecimal string, fieldValue will equal to “10000000″.

And “01D7″ is not a boundary value. So chances are a standard test case will not include it (boundary value in this case is 0000, and FFFF). And this bug will not have revealed itself during normal testing or regression testing.

Exploratory testing may reveal it. If we are lucky.

It was just taking way too long to get a build

Today I faced with one of the very common problem in legacy software development. Getting a build is taking way too long.

The reason for today’s build delay: Merge issues in data files management causing some data lost in the process of merging.

I don’t know what is the best way to tackle this. But legacy code, coupled with manual data management is a disaster. Which is why “build often-continous integration” is hard. Because it forces us to keep the pieces together as closely as possible, rather than just focusing on building the pieces as huge as possible.

If you are in a software development process where the data management in the system is manual and painful, consider improving it. Because your system can have the best design ever, but if your data merging process is long and error-prone, you have an error prone system. Period.

Here is a real world case study.

Previous system: When a new version of the software is ready for release, developers have to manually find the delta between distributed data files and the new data files. Developers create the delta file, include in the install, instruct the install to kick off merging job. So merging data files is usually present in the last few builds in the QA cycle because as your data files changes during many rounds of QA, you probably do not want to redo your manual delta merging many times.

New system: The data file merging is done in code dynamically. During startup, the software calls a routine to detect any delta differences, then calls the merging routine appropriately if needed. Time saved = unbelievable. I have to say, this new system’s improvement come at a cost though. The time to implement the merging is long, but it is a one-off cost that benefits the entire software development cost long term.

I am glad we did the improvement. But the sad part is the old system is still active release so we are still supporting it the manual way for old release (as our new release is not backward compatible).

Less than ideal, but getting better.

Test often, test early: Will it ever fly?

I am very interested in testing software and code because I am a software developer and I writes testing solutions for software companies.

And yes I am very interested, and have been practicing test-driven development in my working place. I believe in it because I think the value in testing is enormous. It is very much heavy lifting, but it makes debugging so much easier later.

Our message simulation tool is just one of those that you have to invest a lot up front for the work, to setup messages specific to the organization. But it pays off when you run many regression tests after that. With continous integration, everytime you have a build, you can run a set of regression tests on it, before you verify it is a stable build for QA.

The value is huge compared to the upfront investment you have to do. But it is hard to quantify financially in terms of ROI. Hours saved by implementing the tool? Well, you have a lot hours to invest into it first.

We can come up with out-of-the-box test scripts that works. But guess what, these scripts will get outdated once a bi-annual mandate comes out from Visa or MasterCard. The better idea behind this is education. Educate managers so they understand the importance of regression tests. Educate them that testing is not the last step in verifying your code works, it is in every step of your development that you know your stuff works.

Test driven development concept has been embraced by many people, but traditional legacy software developers may still need some time to get used to it.

I believe that spreading the concept of testing is just as important as marketing the tool itself. Tool is a supplement if managers understand the concepts.

We will see how far we get.

Build frequent or build when done?

One of the most challenging piece of software development is the integration phase. In the entire software development process: design, coding, integration, testing, coding is the easiest phase. It is because if you allocate time to think through before you write your code, chances are you write code that is of high quality. Integration is the toughest part. Because often time a small part of your change has to work with the entire system that have many third party components that you have no control over, many legacy components that have to be in the system, and many components that you and your team have written separately.

And because of that, many software development veterans advocate continuous integration. It enforces discipline among developers. In Martin Fowler’s article, he said it is communication among developers.

To put a continuous integration in place is difficult, especially you have a huge legacy code base. When integration is a painful phase, you know you have a problem. And in today’s environment when everyone wants things to get done fast and cheap, we tend to defer fixing the problem. That is the real issue. Recognizing a problem but putting it off as if you can move on with that problem unfixed.

In the financial industry, reliability is the key. And to increase high reliability on our code, continuous integration is an essential part. And the testing that is part of the continuous integration process should be automated. It has to be ready to be fired off as soon as the build is complete.

When you put in a new feature in your financial switch to accept EMV messages, you need to be sure your non-EMV transaction still works as before (or as you expected). And you should not need to have to allocate expensive resources/hardware just to send a message through your development switch to regression test your feature. It should be as easy as just firing up a card processor simulation and send out messages to your newly compiled switch. All you need for testing should be just some pre-setup messages.

How often financial institutions, and switches do that? I have not been in this industry long enough to tell. I know they are doing it, but is regression testing the last piece they (dreaded) do, or is it part of their frequent integration process?

Testing is more important than coding, because it tells you if you have broken something that has worked previously. A new feature is nice to have, but existing functionality is critical to have.

It is important to have the most easy-to-use,”high-end” IDE for developers (yes, you really need to setup the right environment to get the productivity level you expect from software developers), but it is equally important you put in place an integration process that includes regression test. It affects the quality of your end product, which your company’s future depends on.

And it is hard to get continuous integration in place especially you have a huge legacy code base that is very difficult to test. But you have to start somewhere. The sooner you can start, the sooner you get over it. Yes it is painful because the team I am working with is going through the beginning of continuous integration. It is painful to automate builds. It is hard, but I think it is worth it.

Really, I think when I went for a computer science degree, I always thought the hardest part of being a software developer is learning to code. With more than two years into this industry, I realize that the hardest part is to manage the process to ensure you have a high quality product. You can have the most coolest product on a most advanced platform technologically, but if your product only works 50% of the time on these cool features, it would not fly in the financial industry. And probably not long as well in other industries.

Same goes with life in general. You can be most talented, but if you are going to excel in the corporate world, you need to learn to work with different people often.

Designing software

Software development is like life. The schedule keeps changing, nothing is set in stone. Requirements changes according to changes in the environment, just like marriage in these modern days.

Software development in the financial industry is even more challenging in a sense. The products are perceived to be highly reliable, because they dealt with money. One of my collegues used to say: if you write the firmware that runs on a router, and if it has bugs that cause you to lose one email, no one really cares. If a financial switch loses one transaction, even if that transaction is only a dollar, a cent, whatever minimal amount that you can think of, EVERYONE cares. It is a big thing.

And here I am today, working in the software engineering side in the financial industry. Sometimes we spent a lot of time to make sure something has to be done right, even if the chances of the bug recurring is minimal. Because in this industry, it is reliability that counts. If you can’t have it perfect, then you better know the cost of damage if the bug does reveal itself. It is like making decisions in every junction of our lives, if you better know the cost if things do not turn out right.