A TestSphere Expansion

Software Testing, TestSphere

Let’s begin with a special thanks to Benny & Marcel. Where would we ever be without the good help of smart people?

BandM.png

Benny & Marcel making a case for Testing in Production


It’s been 2 years since we launched TestSphere: A card deck that helps testers and non-testers think & talk about Testing.
People keep coming up with wonderful ideas on how to further improve the card deck. Expansions, translations, errata,…

A Security deck! A Performance deck! A Usability deck! An Automation deck!
Well… yes. The possibilities are huge, but it needs to make sense too: Value-wise & business-wise.
The thing TestSphere does extremely well is twofold: Spark Ideas and Spark Conversation – Thinking & Talking

Maja

Maja being ‘Business Manager’ for a RiskStorming workshop for DevBridge, Kaunas

RiskStorming is developing to become an incredibly valuable format. It combines the two aspects of TestSphere perfectly.
In its essence it makes your whole team have a structured conversation about quality that is loaded with new ideas and strategies. To be blunt: It helps testers figure out what they are being paid for and it helps non-testers find out why they have testers in the first place.

It’s the learnings and insights of having run continuous RiskStorming workshops for many different businesses in many different contexts that drive the new TestSphere expansion.

The creation of an expansion is driven, not by novelty, but from a clear need.

I present you here the first iteration of all new concepts on the cards. No Explanations or Examples yet. We’ll keep the iterations lean. If you have feedback, you can find me on ‘All the channels’.

Five New Cards Per Dimension

In the first version we had 20 cards per dimension. We noticed that some important cards were missing. The new expansion will cover these.

  • Heuristics: Possible ways of tackling a problem.
    • Dogfooding
    • Stress Testing
    • Chaos Engineering
    • Three Amigo’s
    • Dark Launch

+

  • Techniques: Clever activities we use in our testing to find possible problems.
    • OWASP Top Ten
    • Peer Reviews
    • Mob Testing
    • Feature Toggles
    • Test Driven Development

+

  • Feelings: Every feeling that was triggered by your testing should be handled as a fact.
    • Informed
    • Fear
    • Overwhelmed
    • Excited
    • Unqualified

+

  • Quality Aspects: Possible aspects of your application that may be of interest.
    • Observability
    • Measureability
    • Business Value Capability
    • Scalability
    • Availability

+

  • Patterns: Patterns in our testing, but also patterns that work against us, while testing such as Biases.
    • Single Responsibility Principle
    • Story Slicing
    • Mutation Testing
    • Strangling Patterns
    • Long Term Load testing

+

Two New Dimensions

Dimensions are the aspects of the cards that are divided by represented colors. We felt like some important dimensions were missing. Both of these are mainly operations related, a not to be underestimated part of testing.

Hardening: (working title) Concepts that improve the underlying structures of your software. Compare this dimension to muscle building – You need to strain your muscles until the weak parts get small tears, the tissue can then regenerate and build a stronger, more robust muscle. We test, exercise and strain the product so that we can fill the cracks with smarter ideas, better code and stronger software.

  1. Blameless Post Mortem
  2. Service Level Objectives/Agreements
  3. Anti-Corruption Layer
  4. Circuit Breaker
  5. Bulkhead
  6. Caching
  7. Distributed systems
  8. Federated Identity
  9. Eventual Consistency
  10. API Gateway
  11. Container Security Scanning
  12. Static Code Analysis
  13. Infrastructure as Code
  14. Config as Code
  15. Separation of Concerns
  16. Command Query Responsibility Segregation
  17. Continuous Integration
  18. Continuous Delivery
  19. Consumer Driven Contract Testing
  20. Pre Mortem

Post-Release: (working title) Tactics, approaches, techniques,… that improve the ability to see what’s going on – and orchestrating safe changes in your application’s production environment. When something goes wrong, goes well, brings in money, throws an error, becomes slow,… You can see it and its results.

  1. Fault Injection
  2. Logging
  3. Distributed Tracing
  4. Alerting
  5. Anomaly Detection
  6. Business Metrics
  7. Blackbox Monitoring
  8. Whitebox Monitoring
  9. Event Sourcing
  10. Real User Monitoring
  11. Tap Compare
  12. Profiling
  13. Dynamic Instrumentation
  14. Traffic Shaping
  15. Teeing
  16. On-Call Experience
  17. Shadowing
  18. Zero Downtime
  19. Load Balancing
  20. Config Change Testing

Wrapping up

I’m out of my water here. There’s so much I need to investigate, learn, put into words for myself before I can make it into a valuable tool for you. I welcome any feedback.
Thank you for being such an amazing part of this journey already.

Knowit

The winning team of the RiskStorming workshop at TestIT in Malmö

A PO’s View on Releasing

Uncategorized

… and what that means for the tester.

I’m still in the Healthcare project where we’re strangling small pieces of functionality out of a big monolith of software that was built over the course of 20+ years.
Next month, we’ll put our first real strangled functionality into production. In a docker container. With horrible UI. Yes, releasing is a matter of months for us. That’s not what I’m worried about right now.

The past week, I noticed myself saying: “I don’t care about quality.” and “We don’t need to fix that logical gap.”. Things I wouldn’t be able to reconcile myself with when I was a tester. My view on releasing, and the quality of said release has changed as well.

That ‘I don’t care about quality’ was mainly to make a point. It wasn’t correct. I just care less about certain aspects. The UI is not well structured, black and white and far from appealing. But everything we need is there. Looking at our new screen, any user will curl their upper lip.

Additionally, when you open a (closed) file from last year, you’ll get a blank screen. No user friendly message, apology or workaround. It’s not optimal, but it’s just not important to me and I’ll gladly defend that to anyone.

What does interest me are two very different aspects: Security and Performance.
(This is by the way why RiskStorming is such a great workshop to do in the beginning of such a project. As this information shouldn’t come as a surprise to anybody.)

Positive User Feedback is currently a low priority for me. We’re doing a very minor change when it comes to our users. However, the change touches upon some very important data. Data you don’t want to have stolen. That’s my first priority.
The second one is successfully working closer with our Operations department. The better we can make that collaboration, the easier we’ll have it in the future.
A third priority is making it run smoothly in production at all. If it’s a flop, it’s merely a flip of a parameter to revert.

These kind of decisions don’t make me particularly popular with the testers in the team.
I apologise for this, yet they are the product of careful listening and thinking.

20190117_092706

Skiing with Team members

What does this mean for Testers?

In the aforementioned context, it can be daunting to be a tester. They might feel out of the water concerning expected expertise and they might feel discouraged because many issues are dismissed.

Consider this: I’m currently laying tracks for a train to cross America East to West around the 1860’s. I expect it to be bumpy. I expect troubles. The passengers should have a good chance of arriving at the other side, but in the face of danger, we should be able to turn back. The testers can’t safeguard this project from bandits, explosions, landslides,… They also can’t simulate half of these things or more as they lack expertise, resources,…
However, they can gather information on as many potential points of failure as possible and provide possible alternatives.

They might not know Security Testing or Performance Testing, but they know Risks and have the skills to identify weaknesses.
I expect them to use their words and arguments to do everything in their power to make sure I abandon the project as a whole.
… and have the patience, respect and open mind to accept unpopular decisions made by the PO.
Not that this has been an issue as of late, but I clearly remember my own frustration when faced with what my testers are facing now. I guess it’s never too late for some lessons in humility.

20190119_122942

Reflecting on the last project

Experience Reports, Software Testing

This is a post written by Geert van de Lisdonk about a project he worked 1,5 year on as a Test consultant.99-Geert.png

My last project was in the financial sector. The product we made was used by private banks. Our users were the Investment Managers of those banks. They are called the rock stars of the banking world. Those people didn’t have time for us. We could get little information from them via their secretaries or some meeting that was planned meticulously. And in that meeting only 1 or 2 of our people could be present, to make sure they didn’t spent too much time. Getting specs or finding out what to build and build the right thing was not an easy task. Our business analysts had their work cut out for them but did a very good job with the resources they had. Getting feedback from them was even more difficult. Especially getting it fast so we could change the product quickly. Despite all that, we were able to deliver a working product to all clients. This blogpost is a reflection on what I think I did well, what didn’t do well and what I would have changed if could have done it over.

 

What we did well

Handovers

One of the best things we did, in my opinion, were the handovers. Every time something was developed, a handover was done. This handover consists out of the developer showing what has been created to me, the tester, and the product owner.
This moment creates an opportunity for the PO to verify if the correct thing has been build or point out possible improvements.
As a tester this is a great source of information. With both the developer and the PO present,  all possible questions can be answered. Technical, functional and everything in between can be reviewed and corrected if necessary.

Groomings

Getting the tester involved early is always a good idea. When the Business Analysts had decided on what needed to be made, a grooming session was called to discuss how we could achieve this.
Most of the times there was already some kind of solution prepared by the Product Manager that would suit the needs of several clients. This general solution would then be discussed.

For me this was a moment I could express concern and point out risks. This information would also be a base for the tests I’d be executing.

Teamwork

The team I was in is what I would describe as a distributed team. We had team-members in Belgium, UK and 2 places in Italy. Working together wasn’t always easy. In the beginning most mass communication was done using emails sent to the entire team. This didn’t prove very efficient so we switched to using Microsoft Teams.

There was one main channel which we used the most. Some side channels were also set up that would be used for specific cases. People in the team were expected to have Teams open at all times. This sometimes didn’t happen and caused problems. It took some getting used to, but after a while I felt like we were doing a decent job!

whatsapp image 2019-01-21 at 10.59.45

 

What we could have done better

Retrospectives

When I first joined the team the stand-ups happened very ad-hoc. You could get a call between 9am-3pm or none at all. Instead a meeting was booked with a link to a group Skype conversation. Everybody was now expected to join this conversation at 10am for the stand-up. This was a great improvement! Every sprint we would start with a planning meeting and set out the work we were supposed to do.

But there were also ceremonies missing. At no point in time was there a sprint review or a retrospective. This meant that developers didn’t know from each other what had been finished or what the application is currently capable of.

The biggest missing ritual in my opinion was the retrospective. There was no formal way of looking at how we did things and discussing on how we could improve. Having a distributed team didn’t help here. Also the high pace we were try to maintain made it difficult. But if the PM would have pushed more for this, I think the team could have benefited a lot.

Unit testing

There was no incentive to write unit tests. So there were only a handful of them. Not because the developers didn’t want to. They even agreed that we should write them! There was just nobody waiting for them so they didn’t write them.
There were multiple refactorings of code that could have been improved with unit tests. Many bugs were discovered that wouldn’t have existed if only there were some unit tests written. But since nobody asked for it, and the pace was to high, no time was spent on them.

Less pressure

This project was ran at a fast pace. Between grooming and delivery were sometimes only 3 sprints. 1 for analysis, 1 for development, 1 for testing/fixing/deploying. This got us in trouble lots of time. When during development raised new questions or requirements emerged, there was little time for redirection. Luckily we were able to diminish the scope most of the time, but I still feel we delivered lower quality than we would have liked.

whatsapp image 2019-01-21 at 12.25.25

What I would have done differently

Reporting

Looking back, it was difficult for the PM to know exactly what I was doing. We used TFS to track our work, but it wasn’t very detailed. The stand-ups did provide some clarity, but only a partial message.

My testing was documented in a OneNote on the SharePoint, so he technically could verify what I was doing. Although admittedly this would require a lot of energy from him.
I think he would have preferred pass/fail test cases, but I didn’t deem that feasible with the high pace we were trying to maintain.
In hindsight I could have delivered weekly reports or sprint reports of what was done and what issues were found or resolved. This would would of course take some time at the end of the sprint, that could be an issue. I did look for a decent way to report on my testing but never found a format that suited me.

Fix more bugs myself

We were working CRM Dynamics that was altered to fit the needs of our customers. Both the system and the product were built in such a way that most setting could be altered in the UI. It took me a while to learn how these things worked but managed to resolve bug myself. Sometimes I didn’t know how to resolve them in the UI. I would then take this opportunity and have the developers explain to me how to resolve it next time I encounter something similar.

Since the framework restricted us in some ways, we also made use of a C# middleware to deal with more complex things. The middleware issues were harder for me to resolve so I don’t think I would be able to fix those by myself. The middleware developers being in Italy also complicated things. Pairing on the bug fixes could have taught me a lot. This did happen from time to time, but not frequently enough so I could dive in and sort things out myself.
Additionally, having more insights into the application would have been a nice luxury to have. Through tools such as ‘Dynatrace’, Application Insights,… I could have provided more information to the developers.

whatsapp image 2019-01-21 at 10.59.45 (1)

To summarize

Despite the high pace this project was ran, we still managed to do very good things. The people on the development team were very knowledgeable and taught me a lot. Sure there were some things that I would have liked to change, but that will always be the case. To me the biggest problem was that we didn’t reflect on ourselves. This meant we stagnated on several levels and only grew slowly as a team and a product.
I learned that I value (self-)reflection a lot. More than I previously knew. I started looking for other ways to reflect. At DEWT I got advised to start a journal for myself. This is something I plan on doing for my next project. Currently I have a notebook that I use for all testing related events. Not yet a diary, but a start of this self-reflection habit!
I also learned how I like to conduct my testing and where problems might have risen there. Focus on people and software instead of documentation. I would add some kind of reporting to show of my work. I’ve been looking in good ways to do this report, but am yet to find a format that suits me.
                           

 

How my Testers help Me

Uncategorized

In the beginning of last year, I moved from a Test Consultant to a Product Owner role and I haven’t written much since. Gaining so many insights over that period from: My role change, remote working teams, testing strategy workshops in other companies, Agile transformation of a HealthCare company and much more. It’s been a tumultuous year, but a rollercoaster in learning and insights. The end of the ride is not in sight.

My first insight I wish to explore is how I, as a PO, am dependent on testers: My in-team tester, our automation specialist and the separate team of testers.

In my role, I am mainly concerned about delivering a quality product that people are sufficiently happy with. It shouldn’t be stellar. It doesn’t have to be perfect. It has to be an improvement. 

That’s the context I work in. We’re trying to replace a big monolith of ancient software into smaller pieces of understandable, working software using strangling patterns.

A strangling pattern in short: You identify the boundaries of a piece of functionality, you build a completely, separate duplicate of the functionality and feed the exact same inputs to both pieces of functionality in a test or production environment. Then you monitor the outcomes. When, over time, the outcomes are consistently the same you can replace the old functionality with the new.

This pattern is used in situations where functionality has become so complicated, so bloated that any change can cause a ripple effect across the software destroying just about anything.

20180919_124906-effects

Team Day in the mountains

The Team Tester

Given this situation, I rely highly on the software testers. My team’s tester has been working for the company a lot longer than I have and has a wealth of knowledge when it comes to the application and its history. That’s why me and the team need him to:

As Early As Possible:

  • Warn us about possible points of failure;
  • Inform us about potential risks and their outcomes;
  • List problematic parameters and outliers that we might forget;
  • Talk to & prepare other testers about the changes that are coming.

During Development:

  • Support the team by providing information, good and bad, about the current state of the product;
  • Inform the PO about the progression on achieving of sprint goals or not;

After Development:

  • Support me in handing over the changes to our most direct stakeholders;
  • Hand over the changes to the separate test team so that they can see what actually was finished.

In other words, he’s a bit out guardian angel. Not the kind to swoop in and save the day, but the fairy-god-person type that’s always watching our backs and supports us to overcome problems ourselves.

The Automation Specialist

To be honest, when it comes to automation on the service or UI level, it’s not profitable for us yet. Our specialist has her hands full maintaining old selenium scripts for the big monolith. Voices constantly rise up whether these scripts are worth the investment time and again. For now, we persist.

Next to the maintenance, she’s also involved in pair development with the others in the team, learning about implementation new user stories. She’s involved in pipeline improvements and team discussions about for example: the depth and effort the team wants to put in TDD practices and so on.

With our move to Windows containers for our test systems and hopefully also production, she’ll have her hands full supporting the team in providing checks and information on stability.
For I want them to streamline everything so well, that in the future they can give me a button: a button that I can push at any time, that deploys our current devbranch into production.

20180923_165950

OktoberFest, with the Alps in the background

The Separate Test Team

This is a whole different article that needs to be written. Potentially, this is the point in the Agile Transformation that has had the biggest change and effect on the reliability of our releases of all.

In the past year, I’ve seen our releases go from “Always going horribly wrong, needing four to five hotfixes” to “A release with scheduled hotfixes” to “A stable release with little to no problems.”
These changes can be attributed to changes across the whole development process, but it is a marvel to observe how much more efficient the test team has become.

I saw them, under guidance of great test consultants, abandon most of their documented test scripts and develop mindmaps and checklists. They focused on efficiency, speed and supporting rather than complaining, defending and acting like martyrs.

To Conclude

The testers I work with cover a lot of ground. They focus on results and moving forward, by amplifying the team through information. Information that is actionable and valuable.

It hasn’t always been like this. Together, we’ve come from a very ‘structured’ process where people were working in separate roles on a monolithic software problem, making our situation worse and worse by the day.
What turned it around was:

  • Problems being made visible;
  • Protection being provided by (and from) management;
  • People’s roles being mixed;
  • Teams getting a vision, but otherwise left to their own devices;
  • Relentless positivism;
  • Strong consultants that support with knowledge and learning.

Little change after little change each of us started to see a little light in the darkness. That’s all one needs to keep moving forward. So we joined hands and started in that direction together.

20180919_131139

Part of the team, chilling

A Changing Mindset

Experience Reports, Software Testing

Ever since I left my short stint at the meat factory, I’ve been a Software Testing Consultant for all of my modest career. Until a few months ago, when fate threw me into a Product Owner role.
5 months in, I feel my priorities, my thinking, my mindset… change.

This is not necessarily a good thing, but it is a necessary thing. First, I was Product Owner of Test Automation. But as that team disbanded due to too much overhead for a reasonably small team, I became Product Owner of a 8-headed SCRUM team of developer-architects, a tester, a test automation specialist, a DevOps specialist and soon a new junior developer.

My previous two blog posts were about helping a relatively small team learn more, move them forwards and become confident.
My new role is again different and it’s providing me insights about myself, how I adapt to these dynamics.

Mindset

My mindset has changed drastically. Where I was focused on risks, oversights and possible problems before, I am now looking at ‘good enough’ and going forward with the things ‘that matter’. Because of my Testing background and my now PO role, I realise that those two things are very different for me than other team members. I don’t know the risks well enough, I don’t know the scope too well (as the product is very new to me) and I can only guess at the value our changes bring.

Yet, this doesn’t seem to stop me forming opinions and making decisions.

It frightens me to take steps forward into this vast uncertainty of unknown unknowns knowing that I’m probably on top of the Dunning-Kruger ‘Mount Stupid’.
I caught my self disregarding several risks people mentioned, just because they intervened with my plans…
I have critisised many Product Owners before, when I was a tester, that I could see they had no clue what they were doing or where they were going.

 


I’m beginning to believe that this uncertainty is a big part of the role.
I need a tester to keep my feet on the ground.
I need this done as early as possible.

My priorities lie with keeping the team happy and delivering business value to the stakeholders. Not in risks, maintenance or changes…

Because of that, I’m not thinking of 3 out of 4 types of work.


Four Types of Work

When you find yourself in a situation where you don’t know enough or feel inadequate, start learning, reading and discussing. That’s what I do at least. I needed to ‘up my game’.

One extremely important finding for me were the four types of work featured in ‘The Phoenix Project’: Business Projects, Internal IT, Changes and the highly destructive Unplanned Work.

This connected several frustrations of mine into one model.
My current customer is quite good at pinning down Business Projects. At the very beginning, we do a three-amigo kind of thing where we lay the fundamental vision for the project and immediately try to cut down all the surrounding waste.

Internal IT is handled reasonably, though the responsible people seem to live on a well frequented island. We have two Admins who seem to troubleshoot and fix several major problems a day.

Changes are frequently happening, but are largely unmanaged. I’ve added a blank User Story in our sprints to capture ‘surprise tasks’. This should create a good baseline to see where these change requests come from and how much time they soak up. From there on out we can create procedures to mitigate, ignore, prioritise, escalate… What exactly we’ll do with the data, I don’t know yet, but we’ll have a better idea on how to tackle these changes.

I finally can put into words why I as a tester was often a source of frustration for a Product Owner: Unplanned Work. This type of work disrupts your whole flow, motivation, plans and ultimately, can destroy your project. Call it, bugs, risk, oversights,… it’s everything that suddenly requires someone’s attention and who can therefore no do anything that was planned. It eats your plans. It tears apart your flow and energy. It makes sure people get frustrated.

When Work In Progress is often called the silent killer, Unplanned work is the loud bloody zombie apocalypse that comes to exterminate your project. It terrifies me.
… enter the jolly testers who tell us we forgot about something important.

We just had two sprints torn up by the walking dead. Project management: ‘oh, we forgot to include these highly crucial features that need to be in production by the end of the month.’
Nor I, nor the team, was amused.

A Change in Thinking

A year ago, when I was a tester in this situation I would raise many bugs, make them visible and be loud about the frustrations I could notice in the team.
In similar occasions, I’d have given up and watch the train ride into a wall (again) to then see what we could make of the pieces.

Being in this situation as a Product Owner I try to make the best of the situation. Hope for the best and try to plan for the worst.
As a contingency, I put machinations in place that will bring more insight:

  • We will capture the ‘surprise tasks’ that weigh on the team to manage Changes
  • We will analyse the bugs found after development (and initial testing) from the past 6 months to build a checklist that can help us identify Unplanned Work
  • I need to keep a buffer to allow for Unplanned Work

The data in 1 will be a baseline to come up with certain Change Procedure(s).
From the data in 2, we can build automated checks, monitoring, alerts and ‘have you thought about/talked to X’-checklists for management.

I’m now in a role where I don’t have to be the 20-something-year-old screaming bloody murder anymore. It might sound strange or unfair, but my words have more impact these days. I won’t complain.
This phenomenon has given me the power to actually strategise and bring change while being very obvious about it. I’m not trying to persuade people to follow my ideas anymore. I’m gathering them by being direct.


I want to avert future disasters like we’re now in. I want the team to be on top of things. Maybe in the future, we’ll simulate our own disasters, while we’re still in control. Just for fun. And learning.

fb_img_1526309449076