RiskStorming Experience Report: A Language Shift

Experience Reports, Software Testing, TestSphere
20190227_195530

RiskStorming @AgileActors; Athens, Greece

RiskStorming is a format that helps a whole development team figure out the answer to three important questions:

  • What is most important to our system?
  • What could negatively impact its success?
  • How do we deal with these possible risks?

At TestBash Brighton, Martin Hynie told me about another benefit:

It changes the language

The closer we can bring postmortem language into planning language, the closer we get having healthier discussions around what testing can offer and where other teams play a role in that ability.

Martin was so kind as to write down his experiences for me and allowing me the publish them:


Riskstorming as a tool to change the language around testing

One of the most fascinating things to observe is the language that is highly contextual when others speak of testing. Most specifically:

While a project is ongoing, AND a deadline approaches:

  • The general question around testing tends to be about reducing effort
  • “How much more testing is left?”
  • “When will testing be done?”
  • “How can we decrease the time needed for remaining testing?”
  • “Testing effort is a risk to the project deadline”

Once a project is done, or a feature is released… AND a bug is found in production:

  • “Why was this not caught by testing?”
  • “Testing is supposed to cover the whole product footprint?”
  • “Why did our testplan not include this scenario?”
  • “Who tested this? How did they decide what to test?”

There are two entirely different discussions here going on.

  • One views testing as overhead and process to be overcome… because risk somehow discrete is something to be mitigated. This is false, but accepting uncertainty is a hard leap for project planning.
  • The second is retrospective and viewing any failure as a missed step. Suddenly the pressure is an expectation that any bug should have been tested/caught, and the previous expectations and concerns around timelines feel insignificant, now that the team is facing the reality of the bug in production and the impact on customers and brand.

Riskstorming Experiment

By involving product, engineers and engineering management in RiskStorming questions, we were able to reframe planning in the following manner:

  • Where are the areas of uncertainty and risk?
  • What are the ranges of types of issues and bugs that might come from these areas?
  • How likely are these sorts of issues? Given the code we are touching… given our dependencies… given our history… given how long it has been since we last truly explored this area of the code base…
  • How bad could such an issue be? Which customers might be impacted? How hard could it be to recover? How likely are we to detect it?
  • Engineers get highly involved in this discussion… If such an issue did exist, what might we need to do to explore and discover the sorts of bugs we are discussing? How much effort might be needed to safely isolate and fix such issues without impacting the release? What about after the release?

Then we get to the magic question…

Now that we accept that in fact these risks are real because of trade-offs being made on schedule pressure vs testing (and not magically mitigated…):

If THIS issue happened in production, do we feel we can defend

  • Our current schedule,
  • Our strategy for implementation,
  • Our data, and environments for inspecting our solution,
  • Our decision on what is enough exploration and testing
    when our customers ask: “How did testing miss this?

What was interesting, is that suddenly, we were using the same language around testing before the release, that we only ever used after we released knowing that a bug actually happened in production. We used language around uncertainty. We starting using language around the reality that bugs will emerge. We started speaking around methods to perform the implementation that might help us make better use of testing in order to prioritize our time around the sort of issues that we could not easily detect or recover from.

We started speaking a language that really felt inclusive around shared responsibility, quality and outcomes.

I only have one data point involving RiskStorming… but took a similar approach with another team simply with interviewing engineers, and reporting on uncertainty, building a better sense or reality on trade-offs regarding these uncertainties, and options to reduce uncertainty. It had similar positive outcomes as RiskStorming, however required MUCH more explaining and convincing.

 


Martin Hynie

MartinhynieWith over fifteen years of specialization in software testing and development, Martin Hynie’s attention has gradually focused towards embracing uncertainty, and redefining testing as a critical research activity. The greatest gains in quality can be found when we emphasize communication, team development, business alignment and organizational learning.

A self-confessed conference junkie, Martin travels the world incorporating ideas introduced by various sources of inspiration (including Cynefin, complexity theory, context-driven testing, the Satir Model, Pragmatic Marketing, trading zones, agile principles, and progressive movement training) to help teams iteratively learn, to embrace failures as opportunities and to simply enjoy working together.

A TestSphere Expansion

Software Testing, TestSphere

Let’s begin with a special thanks to Benny & Marcel. Where would we ever be without the good help of smart people?

BandM.png

Benny & Marcel making a case for Testing in Production


It’s been 2 years since we launched TestSphere: A card deck that helps testers and non-testers think & talk about Testing.
People keep coming up with wonderful ideas on how to further improve the card deck. Expansions, translations, errata,…

A Security deck! A Performance deck! A Usability deck! An Automation deck!
Well… yes. The possibilities are huge, but it needs to make sense too: Value-wise & business-wise.
The thing TestSphere does extremely well is twofold: Spark Ideas and Spark Conversation – Thinking & Talking

Maja

Maja being ‘Business Manager’ for a RiskStorming workshop for DevBridge, Kaunas

RiskStorming is developing to become an incredibly valuable format. It combines the two aspects of TestSphere perfectly.
In its essence it makes your whole team have a structured conversation about quality that is loaded with new ideas and strategies. To be blunt: It helps testers figure out what they are being paid for and it helps non-testers find out why they have testers in the first place.

It’s the learnings and insights of having run continuous RiskStorming workshops for many different businesses in many different contexts that drive the new TestSphere expansion.

The creation of an expansion is driven, not by novelty, but from a clear need.

I present you here the first iteration of all new concepts on the cards. No Explanations or Examples yet. We’ll keep the iterations lean. If you have feedback, you can find me on ‘All the channels’.

Five New Cards Per Dimension

In the first version we had 20 cards per dimension. We noticed that some important cards were missing. The new expansion will cover these.

  • Heuristics: Possible ways of tackling a problem.
    • Dogfooding
    • Stress Testing
    • Chaos Engineering
    • Three Amigo’s
    • Dark Launch

+

  • Techniques: Clever activities we use in our testing to find possible problems.
    • OWASP Top Ten
    • Peer Reviews
    • Mob Testing
    • Feature Toggles
    • Test Driven Development

+

  • Feelings: Every feeling that was triggered by your testing should be handled as a fact.
    • Informed
    • Fear
    • Overwhelmed
    • Excited
    • Unqualified

+

  • Quality Aspects: Possible aspects of your application that may be of interest.
    • Observability
    • Measureability
    • Business Value Capability
    • Scalability
    • Availability

+

  • Patterns: Patterns in our testing, but also patterns that work against us, while testing such as Biases.
    • Single Responsibility Principle
    • Story Slicing
    • Mutation Testing
    • Strangling Patterns
    • Long Term Load testing

+

Two New Dimensions

Dimensions are the aspects of the cards that are divided by represented colors. We felt like some important dimensions were missing. Both of these are mainly operations related, a not to be underestimated part of testing.

Hardening: (working title) Concepts that improve the underlying structures of your software. Compare this dimension to muscle building – You need to strain your muscles until the weak parts get small tears, the tissue can then regenerate and build a stronger, more robust muscle. We test, exercise and strain the product so that we can fill the cracks with smarter ideas, better code and stronger software.

  1. Blameless Post Mortem
  2. Service Level Objectives/Agreements
  3. Anti-Corruption Layer
  4. Circuit Breaker
  5. Bulkhead
  6. Caching
  7. Distributed systems
  8. Federated Identity
  9. Eventual Consistency
  10. API Gateway
  11. Container Security Scanning
  12. Static Code Analysis
  13. Infrastructure as Code
  14. Config as Code
  15. Separation of Concerns
  16. Command Query Responsibility Segregation
  17. Continuous Integration
  18. Continuous Delivery
  19. Consumer Driven Contract Testing
  20. Pre Mortem

Post-Release: (working title) Tactics, approaches, techniques,… that improve the ability to see what’s going on – and orchestrating safe changes in your application’s production environment. When something goes wrong, goes well, brings in money, throws an error, becomes slow,… You can see it and its results.

  1. Fault Injection
  2. Logging
  3. Distributed Tracing
  4. Alerting
  5. Anomaly Detection
  6. Business Metrics
  7. Blackbox Monitoring
  8. Whitebox Monitoring
  9. Event Sourcing
  10. Real User Monitoring
  11. Tap Compare
  12. Profiling
  13. Dynamic Instrumentation
  14. Traffic Shaping
  15. Teeing
  16. On-Call Experience
  17. Shadowing
  18. Zero Downtime
  19. Load Balancing
  20. Config Change Testing

Wrapping up

I’m out of my water here. There’s so much I need to investigate, learn, put into words for myself before I can make it into a valuable tool for you. I welcome any feedback.
Thank you for being such an amazing part of this journey already.

Knowit

The winning team of the RiskStorming workshop at TestIT in Malmö

TestSphere, The Making Of

TestSphere

The day was 17th of April, 2015 that I mailed Rosie of Ministry of Testing with an idea I had. The question I posed was whether a Testing Tarot Card deck would be something the community would be interested in.

Rosie had thought of something similar before and wanted to work with me on it.

I wanted to create a tool that is generic enough to suit most contexts and can guide testers to find new and exciting ways to approach the product under test.
I started developing something that eventually took the form of a Testing Tarot set.
– Beren

(italics are snippets from actual conversations.)

Phase 1: A Testing Tarot Card deck. – April, 2015

So cards have two sides:
One side is always printed with one/same logo/design.
The other side is printed with the character.
– Rosie

I had already created a list of concepts that would spark inspiration for your testing and connected this with a Tarot-style character.
This is an extract of how that looked:

list

Spreadsheet Keywords – Character – Image description

The next step was finding someone who could draw beautiful characters and designs for the cards.
After contacting several people, we found out that this would be very, very costly.
So we decided to go with someone from Fiverr.com who we’d heard good things about.


After completing about 20 drawings for us, she took 175$ and then vanished into thin air. We had less than half a card deck in a particular style which we couldn’t build on further.
Rosie had invested that money for nothing.

That was the first blow.site

Phase 2: The App – November, 2015

While waiting for the characters to come in, I had created a website to give a better overview of the card deck. This drove me into playing around with colours, logo, icons and a name.
Asking advice from people around me, I found someone who wanted to build this into an app.

To help inspire you, TestSphere doesn’t just give you an objective.
It offers an elaborate explanation of the objective and
gives several examples of how to test the objective.
– TestSphere pitch

 

At that point, we were a year later already. January 2016.
I hadn’t heard as much from Rosie as she was busy with the many other projects. In retrospect, she was right to do so, because I was still searching.
Even if I didn’t know that about myself at the time.


That same month, I was lucky to join 25 other testers on a peer conference: DEWT 6.
I pitched the concept, the website and the app to them.
They felt it had potential, but that it wasn’t quite there yet.

While they were offering constructive criticism, help, support and ideas, I was feeling demoralized. A stone had formed in my stomach.

That was the second blow.

Phase 3: The Card Game – Februari, 2016

Driving home again, I started to get new ideas. Back to basics. A card deck that wasn’t a fluffy fortune telling game, but a useful tool of learning and knowledge sharing.

Again I rethought the list of concepts, form and design of the card deck.
I introduced different dimensions and investigated more concepts to add to the deck. Real, useful and specific test related concepts that have the potential to get testers passionately talking and thinking.

Here’s an example of 9 cards of the first version of TestSphere: pat-3

happycard3

more-concepts

The focus shifted from fortune telling and test ideas to learning and knowledge sharing.


I pitched the new idea at TestBash Brighton as a 99-second talk and that was the moment it got picked up in earnest.
Rosie wanted to get it ready for TestBash Manchester. Marcel Gehlen tested out the game and offered a boatload of feedback.

In total: We liked the cards very much, we have some ideas how we can integrate them in our team / work and we think they add value. For gaming purposes we wanted more rules. If you ever come up with a stricter gaming rule set we are happy to try that out for you.
– Marcel Gehlen

Phase 4: TestSphere – Oktober, 2016

We further expanded on the cards, with examples that approach the concepts from different angles.
A real designer from MoT, Thomas Harvey, joined in and made it look awesome.

3-feelings23-feelings21

This is a product that can be used in many different ways and taps into your experiences, potential and creativity.
Whether you are an experienced developer or junior tester, this game will have you dig deeper and learn from each other.

TestSphere brings out the potential in you.


Phase 5: The Crowdfunding platform – Januari, 2017

We’ve come this far together. All the people who stood by me, supported me and offered advice and work:

Rosie, Marcel Gehlen, Melissa Eaden, Thomas Harvey, Dwayne Slootmans, Bert Lerno, Ben Van Daele and my wife.

Ministry of Testing have invested £20,000 for design, printing and handling.
In order to make that money back, we’ll need to sell about 1,000 card decks.
At the moment of writing, we’ve sold 220.

You can help us take this story further.
Inform your manager, your development team, your marketing team. Get them excited about TestSphere.

Get it at the Ministry of Testing Store.testsphere-15_1024x1024

Phase 6: ???

The App, revisited?
An extension on the Ministry of Testing Dojo, a great library of real life stories?

All we know is, this is far from over. We’re taking this further!

Consult the TestSphere

TestSphere

Last Friday was TestBash Manchester and I’ve enjoyed it thoroughly.
The talks are inspiring, the organisation is impeccable, facilitators are the friendliest people ever and then there’s the people that attend:

All these speakers, listeners, organizers, attendants,… make you feel right at home.
They are what make every edition of Test Bash legendary.

This time, I got the privilege of meeting a whole ton of them while handing out a new card game Rosie and me put together.

testsphere

 

TestSphere: The tool

TestSphere is an idea that has been worked on for two years, has seen 5 different implementations and eventually came alive as a 100-card card deck.

A hundred cards, each featuring one keyword that has something to do with testing.
This keyword is further explained with:

  • A category:
    • Quality aspects
    • Techniques
    • Heuristics
    • Patterns
    • Feelings
  • A slogan
  • Three examples of how this keyword could impact your testing

That’s it! One hundred cards full of test-vocabulary and inspiration.
Can you already imagine how you could use that?

testsphere

TestSphere: The Ice Breaker

Step 1: Spot a lone tester
Step 2: Walk up to them and draw a random TestSphere Card i.e. “Equivalence Partitioning”
Step 3: Ask: “How has “Equivalence Partitioning” affected your testing? Do you have a story or experience to share about that?

What follows is that person thinking a few seconds and eventually give you an interesting story that is the beginning what possibly is a good discussion on that keyword.
Another thing that could happen is the person not understanding what you mean. This gives you the opportunity to practice explaining your testing.
Either way, you’ll have started a discussion where you can coach, teach and learn at the same time.

Additionally: Just keep the cards on your desk at work. Developers, Business people, Managers who come to your desk will say “Ooh, shiny colours! What’s that?”.
Before you know it, you’re teaching your coworkers a thing or two about testing.

testsphere

TestSphere: The Storytelling Game

Step 1: Find a group of 4 to 8 persons
Step 2: Divide the deck by category (20 cards each)
Step 3: Depending on the experience of the group: reveal one or more cards
Step 4: As soon as one person can think of a story that features all revealed cards he or she knocks on the table
Step 5: Tell the story
Step 6: This person takes the revealed cards as full points
Step 7: Other people can also tell their stories to get unrevealed cards for half points.

The person that gets X points first or most points by X time wins.
Easy right? You’ve just gotten a whole group of people thinking deeply about their previous testing experiences and put those experiences to verse.

Additionally: After or even during the game identify opportunities for coaching and helping the others. Point them to resources or people who might help them get more insight.testsphere

TestSphere: The RiskStorming Game

The RiskStorming game is a visual, collaborative method to map your Test Strategy on threats to the quality aspects that matter.
The game consists of three phases:

  1. Select a subset of important quality aspects, blue TestSphere cards. (The things that matter)
  2. Come up with risks to the selected quality aspects. (Threats to the things that matter)
  3. Use the rest of TestSphere to create a strategy in function of those risks. (Test for the threats that impact what’s important)

It is explained much more in depth here: http://thatsthebuffettable.blogspot.be/2017/11/riskstorming-maping-risks-with.html

Also, at the bottom of that blogpost are printable boards for RiskStorming!

 

IMG_20171025_210001.jpg

TestSphere: The Unblocker

So you find yourself bored, blocked, sad or stuck?
The best thing you can do at those times is learn something new.

Draw a random card.
Can this idea infuse your testing in a new way?

  • Have you tried the “too many” heuristic?
  • Have you tested the “Accessibility” Quality Aspect?
  • Could you try doing some “Pair Testing” Techniques?
  • Try exploring your product by applying “force” in creative ways.
  • How would an “Irritated” or “Angry” user use your application?

Additionally: If you don’t know the word on the card, or feel you don’t know if well enough: try to find more opportunities of learning about the concept!

testsphere

TestSphere: Unlimited Possibilities

Job interviews, Brainstorming, Lean Coffee, Analysis tooling, Visual connecting concepts together, Exploring opportunities for personal growth, Storytelling without using keywords, Generating random Test Persona, Re-categorizing the whole thing and connecting the words using different logic,…


TestSphere has been deliberately kept free from rules. I believe testers are creative enough to find use cases on their own and decide for themselves which ones are valuable and which are not.

The only constant is learning from stories and experiences. From your own and from those around you.

We’re not quite ready for mass distribution. We’re still feeling out the market.
However, there will be an option to pre-order one or more decks in the future.

Be sure to hear about it at: @TestSphere

cvxoqo7xeaanmez