Testing Dynamics CRM - The Discipline | Capsule Series

Software is developed by humans, whose propensity for making mistakes is undeniable. Catching those mistakes is paramount to delivering quality solutions to eager customers. Testing is the activity of squashing those bugs. It is an integral part of the software development life-cycle, without which users/customers would be our human Guinea pigs; customer's trust and satisfaction would take a tumble, and projects would probably fail miserably. The first part of this article will go over testing in general. I won't go into too much detail, as this is not the goal of the article. Specifics of Dynamics CRM testing will be covered in later parts of the article.

Is it worth it?

We tend to build in pieces. Imagine building a car, creating the body, wheels, engine, brakes ... etc. Only to discover, after going to production, that customers are complaining about measly acceleration. At that stage, you have to recall and discard the whole lot, make adjustments, and start again -- a 100% profit loss. Testing every aspect of the product before even building a single car could have been avoided such a massive hit; test the engine alone under the expected load. One National Institute of Standard Technology study found that without testing, a 5 hour-fix of a defect in development can take up to 3 times as much if discovered in production.

Prove it!

Some might get away with ship-stopper issues on their blog pages, informational web sites, or even open source small non-financial projects. The most damage incurred would be a loss of a few readers, visitors, or users. Enterprise applications, however,  don't have it so easy. For example, some of Amazon’s third party retailers saw their product price reduced to 1p due to a software glitch, back in 2014 (link). They were left with heavy losses, and some businesses suffered up to a 100,000-Pound loss. Real-time applications' testing criticality could mean the difference between life and death, in some cases. For example, in the 80s, a radiation therapy machine delivered a lethal dose to six patients, causing the death of 4. Investigators discovered that a software bug was the the main cause of the incident. Going a step further to take advantage of testing in preventing bugs from ever entering code in the first place, might allow for software maintenance in half the time at half the cost, as one U.S. Department of Defense study found. We will cover this later in the article. In short, testing properly opens the door to fast, more frequent, and accurate releases.

Great, but ...

... there is a catch to proper and effective testing:

Tight Schedule

Testers could go over every piece of the software by hand, running every scenario, on every device, and in every context manually, or they could write automated test cases. In either case, it often consumes too much time that it could create a bottleneck for the entire pipeline.

Yawn!

Testing benefits are not always clear at first glance, and some stakeholders might feel that it's a drag, as it takes a long-term vision to justify the cost. In addition, most developers find testing mundane, and customers might not appreciate it, considering they don't see the value outright. It takes discipline to stick to a proper tester's mindset.

Regression

Humans are prone to mistakes, lapses, or slipping on repetitive tasks, which leads to overlooking issues and re-introduced bugs when retesting. This could mostly be solved by writing automated testing, but team communication is paramount to maintaining a robust test suite.

Laying the Foundation

Now that we have established the importance and annoyances of testing, any professional tester, and even developers, must know the following fundamental testing principles:

1. Exhaustive testing is not possible
   Most customers I have met expect a "bug-free" solution, whose trust is lost trust gradually with every issue popping up. Regardless of customer's expectations in this area, there's an art in outweighing the risk vs the reward of "seeking" exhaustive testing for the sake of a successful project. Targeting likely sources of the most severe cause of issue for testing could striking the golden balance in this case.
2. Defect clustering
   The Pareto Principle states that about 80% of the problems are found in 20% of the modules.
3. Pesticide Paradox
   Using the same pesticide over and over would lead to insects developing resistance, and eventually the pesticide becoming useless. Same goes for tests: running the same tests repeatedly reduces their effectiveness in discovering new issues; existing tests need to be reviewed and new tests need to be developed to catch new bugs.
4. Testing shows the presence of defects
   Testing reduces the probability of undiscovered issues, but it doesn't prove correctness.
5. Absence-of-errors fallacy
   A 99% bug-free software might still not satisfy the customer's needs. Validation is an essential part of testing where the tester validates the software functionality with the customer's needs. This might be in the form of a UAT testing activity, for example.
6. Early testing
   The earlier the testing starts, the cheaper it is to fix. Testing could start as early as the requirements phase.
7. Testing is context dependent
   Different software require different test steps, procedure, methodologies ... etc. For example, testing a safety-critical system is different from an e-commerce website.

Now that we have gone through some quick general discussion, in the next part of the article, we will breakdown the different parts of CRM and why they are a source of headache when it comes to testing.