Hitachi Vantara Pentaho Community Wiki
Child pages
  • 08. Open Scrum - Hardening and Acceptance Estimation
Skip to end of metadata
Go to start of metadata

As mentioned above the Scrum methodology does not specifically include a hardening phase and so Open Scrum introduces a new step to review evidence that any one feature has reached sufficient maturity.
There are several ways to estimate how hardened a particular feature is. Metrics for measuring hardening and defining acceptance criteria are largely application specific. For instance the specific metric used to assess the hardening of a formula within a SQL database is likely to be different to the metric used to judge the hardening of an dynamic HTML menu-bar .

Download Counts and Forum Posts

This is a widely used, but somewhat flawed, method. This is a passive measurement that assumes:

  1. Widening the window of opportunity for downloading increases the number of downloads.
  2. Increasing the number of downloads increases the number of people using the software.
  3. Increasing the number of people using the software increases the number of use cases applied to the software.
  4. Increasing the number of use-cases applied to the software tests more of the logic within the software and finds more defects if they exist.
  5. Community members are, over time, uniformly likely to report defects that they find.
  6. Community members are, over time, uniformly likely to use any one feature of the software.

For example:

  • A certain feature is implemented in a Sprint and is released to the community in Milestone 1 of the software. Milestone 1 is downloaded 30,000 times. Subsequently 22 bugs are reported on the forums against this feature.
  • Fixes are attempted for the 22 defects and are released in Milestone 2 of the software. Milestone 2 is downloaded 20,000 times. Subsequently 8 bugs are reported.
  • Fixes are attempted for the 8 defects and are released in Milestone 3 of the software. Milestone 3 is downloaded 25,000 times. No further bugs are reported.

It seems reasonable to assume that the quality of the feature has improved. However there is no guarantee that any of the 30,000 downloads of Milestone 3 resulted in anyone using the feature in question. The six assumptions above are individually reasonable over a significant period of time, however applying them together to one months worth of data introduces highly significant variability.

  • Advantages: This method does not require visibility into the use cases or usage of the software by the community and so it can be done passively.
  • Disadvantages: This method only reaches an anecdotal level of accuracy. Information about platform coverage is unknown.

As stated before there are is lot of useful positive feedback that is typically not collected and the passive method above suffers from this.

Positive Feedback Enablers

For the purposes of this discussion I will use the name Positive Feedback Enabler (PFE) to cover embedded tech-niques that are used to increase the amount of positive feedback. These techniques include Heartbeat, Phone-Home, Version-Checker, Acceptance-Check, and Test Suite.

Heartbeat

This is a simple signal that is sent from the software to one of the project's servers to indicate that the software is running. The heartbeat does not send any granular or complex data. When combined with the method above the heartbeat removes the first two assumptions and replaces them with a better estimate of the number of installations running the software.

  • Advantages: Can be done in conjunction with Downloads / Forums. Information gathered is not questionable in nature. Some platform coverage (operating system) can be determined from the weblog of the incoming heartbeats. The community members do not have to take manual actions.
  • Disadvantages: Usage of the software is still not known. Requires infrastructure to accept the heartbeats from the software. Without knowing which version of the software is running educated guesses must still be used to estimate hardening.

This method does not determine whether a given feature is used by anyone.

Version Checker

This is a call from the software to one of the project's servers to determine if a newer version of the software is available. By its very nature a version-checker is an extension of a heartbeat. This method can be combined with any of the other methods. The Version-Checker removes the first two assumptions and replaces them with a quantitative measure of the number of installations running the software.

  • Advantages: Can be done in conjunction with Downloads / Forums. Information gathered is not questionable in nature. Some platform coverage (operating system) can be determined from the weblog of the incoming calls. The community members do not have to take manual actions. It helps to focus community contributions on the latest version of the software and so increases momentum.
  • Disadvantages: Usage of the software is still not known. Requires infrastructure to accept the calls from the software.

This method does not determine whether a given feature is used by anyone.

Acceptance

This is an acknowledgment from the community members that the software is sufficient to meet the needs of their use case in their environment. The software could allow a voluntary description of the use case or other voluntary to be sent as well. This method requires participation from the community

  • Advantages: Can be done in conjunction with Downloads / Forums. Information gathered is not questionable in nature. Some platform coverage (operating system) can be determined from the weblog of the incoming calls. Non-structured feedback can be gathered.
  • Disadvantages: Usage of the software is still not known. Requires manual action by community members. Requires infrastructure to accept the calls from the software. This method does not determine whether a given feature is used by anyone.

Test Suite

This is a test suite that tests the operations of the software in the user's environment and reports the results to one of the project's servers. This method requires participation from the community. The user should be able to review the information before it is sent and/or the software should log the information sent for reviewing later.
This method requires participation from the community. It assumes that:

  1. If a feature does what the user needs it to (meets the needs of their use case without signs of defects)
    and
  2. The feature executes in the user's environment exactly as the developer intended it to
    then the feature is 'ready'.

This method requires that a testing suite is provided with the software. The testing suite needs to have a high level of coverage to ensure that the tests prove the software executes exactly as intended. This method also requires that the user has a way to submit the test results to the project administrators along with (optionally) platform information (operating system etc) and an acknowledgment that the software meets their needs without defects.

The optional information obtained can be used to help prioritize future work. For example if the core developers are all using Microsoft's Windows or Sun's Solaris OS but the submissions indicate that the majority of the community are using Linux or Apple's OS X it should encourage the developers to be more inclusive in their choice of development environment.

  • Advantages: Can be done in conjunction with Downloads / Forums. Information gathered is not questionable in nature. Platform coverage is known.
  • Disadvantages: Usage of the software is still not known. A community member is not inherently self-motivated to provide hardening feedback so an open source project will have to appeal to or motivate its community to participate in this manner. Requires deployment of test suite with significant code coverage and a method for execution. Requires infrastructure to accept the submissions from the community. This method does not determine whether a given feature is used by anyone.

We stated above that the community is not inherently self-motivated to provide hardening feedback. I believe that motivation does exist if there is a clearly documented and causal relationship between submitting acceptance / hardening feedback and versions of the software being declared 'ready' faster:

  1. Community members who are waiting for the 'declared ready' version of the whole release can help the project administrators to make the determination happen sooner.
  2. Community members who are waiting for features that are coming in the next release can help the devel-opment of those features start sooner.
  3. Community members who are waiting for individual features to be 'declared ready' so that they can work on localizations can help make the determination happen sooner.
  4. Potential community members who do not have other ways to contribute because they have not found any bugs, or documentation issues, or platform-specific issues etc are provided with a mechanism for participating.

Usage Metrics

This method involves adding usage tracking to the software. Usage tracking could use standard technologies such as a logger. The usage tracker records when certain locations in the logic are reached. These points are determined by the Product Owner.
Usage trackers fall into two categories:

  • Feature: These are high-level, permanent trackers that are used in aggregation to help refine the Product Backlog by identifying features in the software that are popular and heavily used.
  • Granular: These are more detailed, temporary trackers used only during the hardening phase. As features are determined to be hardened granular trackers should be removed from the hardened code and new ones added to newer features.

The usage tracker should not violate any of the freedoms that are expected of open source software. The usage tracker should not log any parameters or data that is passed or do anything to determine the purpose for which the software is being used. For instance it would be Ok for a chart engine to determine whether anyone ever uses the bubble chart feature. It would not be Ok for it to determine that one user is using the chart engine in a search for an end to world hunger whereas another user is using it to analyze the effects of bleach on fluffy white baby rabbits. The usage tracker should not submit any information without the acceptance of the user.

If this method seems intrusive or shocking here are some mitigating factors:

  • In many cases the information is generated already. If you look at the debug log generated by a lot of software the information that we are talking about sharing is a small subset of this information.
  • Submission of this information will be voluntary.
  • Submission of this information will be a useful contribution to the project. In many cases when the software meets someone's use case without any defects, there is no immediately apparent easy way that the person can contribute to the project.
  • Advantages: Can be done in conjunction with both the methods above. Usage is known. Platform coverage is known.
  • Disadvantages: The nature of the information gathered could be questioned. A community member is not inherently self-motivated to provide hardening feedback so an open source project will have to appeal to or motivate its community to participate in this manner.
  • No labels