Reflecting on the last project

Experience Reports, Software Testing

This is a post written by Geert van de Lisdonk about a project he worked 1,5 year on as a Test consultant.99-Geert.png

My last project was in the financial sector. The product we made was used by private banks. Our users were the Investment Managers of those banks. They are called the rock stars of the banking world. Those people didn’t have time for us. We could get little information from them via their secretaries or some meeting that was planned meticulously. And in that meeting only 1 or 2 of our people could be present, to make sure they didn’t spent too much time. Getting specs or finding out what to build and build the right thing was not an easy task. Our business analysts had their work cut out for them but did a very good job with the resources they had. Getting feedback from them was even more difficult. Especially getting it fast so we could change the product quickly. Despite all that, we were able to deliver a working product to all clients. This blogpost is a reflection on what I think I did well, what didn’t do well and what I would have changed if could have done it over.

 

What we did well

Handovers

One of the best things we did, in my opinion, were the handovers. Every time something was developed, a handover was done. This handover consists out of the developer showing what has been created to me, the tester, and the product owner.
This moment creates an opportunity for the PO to verify if the correct thing has been build or point out possible improvements.
As a tester this is a great source of information. With both the developer and the PO present,  all possible questions can be answered. Technical, functional and everything in between can be reviewed and corrected if necessary.

Groomings

Getting the tester involved early is always a good idea. When the Business Analysts had decided on what needed to be made, a grooming session was called to discuss how we could achieve this.
Most of the times there was already some kind of solution prepared by the Product Manager that would suit the needs of several clients. This general solution would then be discussed.

For me this was a moment I could express concern and point out risks. This information would also be a base for the tests I’d be executing.

Teamwork

The team I was in is what I would describe as a distributed team. We had team-members in Belgium, UK and 2 places in Italy. Working together wasn’t always easy. In the beginning most mass communication was done using emails sent to the entire team. This didn’t prove very efficient so we switched to using Microsoft Teams.

There was one main channel which we used the most. Some side channels were also set up that would be used for specific cases. People in the team were expected to have Teams open at all times. This sometimes didn’t happen and caused problems. It took some getting used to, but after a while I felt like we were doing a decent job!

whatsapp image 2019-01-21 at 10.59.45

 

What we could have done better

Retrospectives

When I first joined the team the stand-ups happened very ad-hoc. You could get a call between 9am-3pm or none at all. Instead a meeting was booked with a link to a group Skype conversation. Everybody was now expected to join this conversation at 10am for the stand-up. This was a great improvement! Every sprint we would start with a planning meeting and set out the work we were supposed to do.

But there were also ceremonies missing. At no point in time was there a sprint review or a retrospective. This meant that developers didn’t know from each other what had been finished or what the application is currently capable of.

The biggest missing ritual in my opinion was the retrospective. There was no formal way of looking at how we did things and discussing on how we could improve. Having a distributed team didn’t help here. Also the high pace we were try to maintain made it difficult. But if the PM would have pushed more for this, I think the team could have benefited a lot.

Unit testing

There was no incentive to write unit tests. So there were only a handful of them. Not because the developers didn’t want to. They even agreed that we should write them! There was just nobody waiting for them so they didn’t write them.
There were multiple refactorings of code that could have been improved with unit tests. Many bugs were discovered that wouldn’t have existed if only there were some unit tests written. But since nobody asked for it, and the pace was to high, no time was spent on them.

Less pressure

This project was ran at a fast pace. Between grooming and delivery were sometimes only 3 sprints. 1 for analysis, 1 for development, 1 for testing/fixing/deploying. This got us in trouble lots of time. When during development raised new questions or requirements emerged, there was little time for redirection. Luckily we were able to diminish the scope most of the time, but I still feel we delivered lower quality than we would have liked.

whatsapp image 2019-01-21 at 12.25.25

What I would have done differently

Reporting

Looking back, it was difficult for the PM to know exactly what I was doing. We used TFS to track our work, but it wasn’t very detailed. The stand-ups did provide some clarity, but only a partial message.

My testing was documented in a OneNote on the SharePoint, so he technically could verify what I was doing. Although admittedly this would require a lot of energy from him.
I think he would have preferred pass/fail test cases, but I didn’t deem that feasible with the high pace we were trying to maintain.
In hindsight I could have delivered weekly reports or sprint reports of what was done and what issues were found or resolved. This would would of course take some time at the end of the sprint, that could be an issue. I did look for a decent way to report on my testing but never found a format that suited me.

Fix more bugs myself

We were working CRM Dynamics that was altered to fit the needs of our customers. Both the system and the product were built in such a way that most setting could be altered in the UI. It took me a while to learn how these things worked but managed to resolve bug myself. Sometimes I didn’t know how to resolve them in the UI. I would then take this opportunity and have the developers explain to me how to resolve it next time I encounter something similar.

Since the framework restricted us in some ways, we also made use of a C# middleware to deal with more complex things. The middleware issues were harder for me to resolve so I don’t think I would be able to fix those by myself. The middleware developers being in Italy also complicated things. Pairing on the bug fixes could have taught me a lot. This did happen from time to time, but not frequently enough so I could dive in and sort things out myself.
Additionally, having more insights into the application would have been a nice luxury to have. Through tools such as ‘Dynatrace’, Application Insights,… I could have provided more information to the developers.

whatsapp image 2019-01-21 at 10.59.45 (1)

To summarize

Despite the high pace this project was ran, we still managed to do very good things. The people on the development team were very knowledgeable and taught me a lot. Sure there were some things that I would have liked to change, but that will always be the case. To me the biggest problem was that we didn’t reflect on ourselves. This meant we stagnated on several levels and only grew slowly as a team and a product.
I learned that I value (self-)reflection a lot. More than I previously knew. I started looking for other ways to reflect. At DEWT I got advised to start a journal for myself. This is something I plan on doing for my next project. Currently I have a notebook that I use for all testing related events. Not yet a diary, but a start of this self-reflection habit!
I also learned how I like to conduct my testing and where problems might have risen there. Focus on people and software instead of documentation. I would add some kind of reporting to show of my work. I’ve been looking in good ways to do this report, but am yet to find a format that suits me.
                           

 

A Changing Mindset

Experience Reports, Software Testing

Ever since I left my short stint at the meat factory, I’ve been a Software Testing Consultant for all of my modest career. Until a few months ago, when fate threw me into a Product Owner role.
5 months in, I feel my priorities, my thinking, my mindset… change.

This is not necessarily a good thing, but it is a necessary thing. First, I was Product Owner of Test Automation. But as that team disbanded due to too much overhead for a reasonably small team, I became Product Owner of a 8-headed SCRUM team of developer-architects, a tester, a test automation specialist, a DevOps specialist and soon a new junior developer.

My previous two blog posts were about helping a relatively small team learn more, move them forwards and become confident.
My new role is again different and it’s providing me insights about myself, how I adapt to these dynamics.

Mindset

My mindset has changed drastically. Where I was focused on risks, oversights and possible problems before, I am now looking at ‘good enough’ and going forward with the things ‘that matter’. Because of my Testing background and my now PO role, I realise that those two things are very different for me than other team members. I don’t know the risks well enough, I don’t know the scope too well (as the product is very new to me) and I can only guess at the value our changes bring.

Yet, this doesn’t seem to stop me forming opinions and making decisions.

It frightens me to take steps forward into this vast uncertainty of unknown unknowns knowing that I’m probably on top of the Dunning-Kruger ‘Mount Stupid’.
I caught my self disregarding several risks people mentioned, just because they intervened with my plans…
I have critisised many Product Owners before, when I was a tester, that I could see they had no clue what they were doing or where they were going.

 


I’m beginning to believe that this uncertainty is a big part of the role.
I need a tester to keep my feet on the ground.
I need this done as early as possible.

My priorities lie with keeping the team happy and delivering business value to the stakeholders. Not in risks, maintenance or changes…

Because of that, I’m not thinking of 3 out of 4 types of work.


Four Types of Work

When you find yourself in a situation where you don’t know enough or feel inadequate, start learning, reading and discussing. That’s what I do at least. I needed to ‘up my game’.

One extremely important finding for me were the four types of work featured in ‘The Phoenix Project’: Business Projects, Internal IT, Changes and the highly destructive Unplanned Work.

This connected several frustrations of mine into one model.
My current customer is quite good at pinning down Business Projects. At the very beginning, we do a three-amigo kind of thing where we lay the fundamental vision for the project and immediately try to cut down all the surrounding waste.

Internal IT is handled reasonably, though the responsible people seem to live on a well frequented island. We have two Admins who seem to troubleshoot and fix several major problems a day.

Changes are frequently happening, but are largely unmanaged. I’ve added a blank User Story in our sprints to capture ‘surprise tasks’. This should create a good baseline to see where these change requests come from and how much time they soak up. From there on out we can create procedures to mitigate, ignore, prioritise, escalate… What exactly we’ll do with the data, I don’t know yet, but we’ll have a better idea on how to tackle these changes.

I finally can put into words why I as a tester was often a source of frustration for a Product Owner: Unplanned Work. This type of work disrupts your whole flow, motivation, plans and ultimately, can destroy your project. Call it, bugs, risk, oversights,… it’s everything that suddenly requires someone’s attention and who can therefore no do anything that was planned. It eats your plans. It tears apart your flow and energy. It makes sure people get frustrated.

When Work In Progress is often called the silent killer, Unplanned work is the loud bloody zombie apocalypse that comes to exterminate your project. It terrifies me.
… enter the jolly testers who tell us we forgot about something important.

We just had two sprints torn up by the walking dead. Project management: ‘oh, we forgot to include these highly crucial features that need to be in production by the end of the month.’
Nor I, nor the team, was amused.

A Change in Thinking

A year ago, when I was a tester in this situation I would raise many bugs, make them visible and be loud about the frustrations I could notice in the team.
In similar occasions, I’d have given up and watch the train ride into a wall (again) to then see what we could make of the pieces.

Being in this situation as a Product Owner I try to make the best of the situation. Hope for the best and try to plan for the worst.
As a contingency, I put machinations in place that will bring more insight:

  • We will capture the ‘surprise tasks’ that weigh on the team to manage Changes
  • We will analyse the bugs found after development (and initial testing) from the past 6 months to build a checklist that can help us identify Unplanned Work
  • I need to keep a buffer to allow for Unplanned Work

The data in 1 will be a baseline to come up with certain Change Procedure(s).
From the data in 2, we can build automated checks, monitoring, alerts and ‘have you thought about/talked to X’-checklists for management.

I’m now in a role where I don’t have to be the 20-something-year-old screaming bloody murder anymore. It might sound strange or unfair, but my words have more impact these days. I won’t complain.
This phenomenon has given me the power to actually strategise and bring change while being very obvious about it. I’m not trying to persuade people to follow my ideas anymore. I’m gathering them by being direct.


I want to avert future disasters like we’re now in. I want the team to be on top of things. Maybe in the future, we’ll simulate our own disasters, while we’re still in control. Just for fun. And learning.

fb_img_1526309449076

Product Owner of Test Automation 2

Experience Reports, Software Testing

In my previous post, I explained the strategy I envisioned for the team. Comparing it to a boardgame.

What this post lacked thoroughly, was a clear focus on team learning. I feel like an idiot for not noticing this earlier.

Learning Objectives as part of the sprint

What has become abundantly clear to me is that the Team Members are the heart of your team. They need to be nurtured, grown.
In our team, we’ve been actively investing into people to become more confident and knowledgeable. ‘Learning objectives’ have become about 50% of our sprint stories.

I add in: Spikes, Proof of Concepts, Blog Posts, Challenges,… to have people work through material and produce reports, concepts, demo’s or anything that reproduces the acquired knowledge. After that, they ask feedback from other team members, discuss or teach. The aim is to achieve two things: new learnings and something valuable for the team, project or product. This keeps our stakeholders happy and our team in learning-mode.

But 50% is a lot of time… how do you explain this to stakeholders?
Test Automation is a valuable endeavor. Though in uncertain conditions it can be rendered useless, time consuming or time wasting even. That’s where we are now with the team. Many different things are changing. The application, the architecture and the development teams are all getting a good shake. This is not a good moment to heavily invest in UI or even API checks. Instead, I shift the focus of the team in a different direction.

Whereto now?

I see a lot of opportunities to coach, train and pioneer automation strategies, as a team.
Once the dust settles from all the management decision-making and architecture workshops it’ll fall on the automation people to strongly improve our release pipeline.
To achieve this, we need to become better at what we do, need to become more confident in what we say and become more respected for the value we bring.
As a team.

Instead of building more automation, our focus shifts to coaching, training and knowledge sharing. The issue, however, is that we first need to do knowledge gathering and train ourselves. The good thing is, we’re more of a team now than before and we can help each other out. We also have some time to invest in ourselves, which will pay off tenfold in the future. Hopefully.

In congruence to the team building up their skills, I’m monitoring progress of these changes and looking for opportunities to help out. Whether it’s now or in the future, I want to know where we can add business value fast. Additionally, I’m collecting examples of good practices in our context and using those as a basis to build an automation strategy.

Changes are coming our way, but we’re preparing to deal with them.

20180214_153619

The Automation team, at sprint kickoff

Applied heuristics

Experience Reports, Software Testing

This is an experiment. I’m trying to figure out my understanding of what the word ‘Heuristics’ means to me and whether it’s helpful to me and my craft: Testing.

Therefore, I’ll rehash an old story of how I found a bug and annotate the heuristics used in Bold and see what I can analyse from that afterwards.

Any guidance, ideas, intuitions, sources are very much appreciated.


Finding a Memory Leak

It was past noon. I’d been testing an application that wasn’t too complex for quite a long time. I knew it’s ins and outs and I felt myself getting bored.
I remember going through the customer files rather methodically one by one, click,… click,… click , without a clear aim. You could say I was randomly exploring.

Until something didn’t feel right. Call it surprise, intuition, something was wrong.
I noticed a pattern of loading times slowing down almost unnoticeable. I can’t put to words what triggered it for me. Was it an Observer-expectancy effect? Was it my negative mindset that sharpened my senses? I haven’t got a clue, but I felt I had to focus, zoom in.

I chose a tactic that was less boring than meticulously clicking my way up. Selenium IDE isn’t the best of tools, but it fit my purpose perfectly. I recorded my click and copied it a thousand times. I monitored the behaviour with developer tools, pressed ‘play’ and went for a coffee. After a while, I could identify that it went terribly slow because I could see the latency increase. Eventually I saw it ramping up until it crashed. Pairing with a developer we could conclude that there was a serious memory leak.


Implicit and Explicit heuristics?

A heuristic is “A fallible approach to solve a problem”

The teachings of RST and BBST and possibly how ‘heuristics’ were meant in the science books would identify all the bold phrases as some variation of a heuristic. I’m sure many other heuristics played in my mind that I don’t have words for yet.

However, notice how the first part of my story is almost completely based on hunches and feelings. Something directed me to find this bug and it wasn’t intentional.

In the second part, I took matters into my own hand. I had a goal, I chose a tactic and was able to measure my results. Other tactics, could have been clicking the button myself for the umpteenth time and miss out on a coffee. I might have asked a developer to look into it straight away and maybe had time left for two or three coffees.

I can’t know for sure whether I chose the correct heuristic or tactic for that situation and I’m sure as hell “# of cups of coffee drunk” isn’t the correct way to measure that success.

We can’t go back in time and compare results of heuristics used vs. the ones not used. Though I’m pretty happy with the results of the one I set up.


The Question Remains

Are the ‘Implicit heuristics’, (i.e. feelings, patterns, biases, hunches) which can’t really be controlled, measured,… useful to call heuristics? They fail, yes. They help you notice things, yes. But do they solve problems?

Are ‘Explicit heuristics’, (i.e. repetition to find boundaries, monitoring for abnormalities, pairing to increase understanding) worth a dime without their implicit counterparts?

 

It’s not up to me to answer these questions. Yet I find it intriguing to ponder over the matter.

 

IMG_20170718_102211846

The BBST Foundations course: Week 4

Experience Reports, Software Testing

The Final Week

It’s the monday of the last week of BBSt, I flunked the last assignment and that had angered me.
In fact, this released a lot of the frustrations I had towards the course in one moment.

I had a short discussion with the instructor and this cleared up a few things. I hadn’t quite understood in what way I had to explain my answer and the instructor hadn’t found what I wanted to say.

We had a long google-hangout session and cleared a lot of things out. Apparently, there were a few videos and pages I missed that were key to answering the exam successfully.

For example, there’s a list of keyword. If the question contains “List X” you give 3 examples of that list. Number 4 and number 5 will be ignored, unless they contain errors; in that case, they’ll subtract marks.
Another, If it says “Describe”, you have to paint a picture. “Describe the Weibull curve” becomes: “A fast surge in the beginning, a flattening until it reaches the peak and then a deep plummet down until its pace declines and steadily, but slowly falls down to 0.

So yes, you need to know these things to be successful in the course. No, it has nothing to do with testing, apart from the fact that “precision reading” is a core skill of a tester.

I eventually got to fill out all the exam questions and discussed several answers of the other students.
I tried to be everywhere and discuss everything worth discussing.
In the end, there was a lot less activity this week than all the others.


The Exam

The exam was a three day, closed book exam. The instructors count on your honour not  to cheat. But it’s really easy to cheat. Really, really easy…. And we’re testers.
Testers cheat.

I had everything stored locally. All my answers, all other people’s answers, all the quizzes…
My book is full of post-its with all the definitions and important information on it.

I really like cheating, I do.
Yet somehow, I was able to fight the temptation. My honour is unscathed. This is probably because I didn’t really need it. I had answered every question already before and I had done this meticulously. I was pretty confident in my answers.
Apart from that, during the exam, you experience sparks of brilliance. You think of things you weren’t able to before.

Gabi, the instructor, had told me that might happen. I didn’t believe it, but paid attention to it none-the-less. He was right.

After the Exam

Ru, another instructor, and me went over my exam questions in a Skype meeting. She had lots of feedback and gave me an appreciation for my answers.
Even-though there was a question in the pool which I had answered similarly wrong as the practice question, Ru gave me the chance to defend and change my initial answer.

The conclusion in the end was:

  • I have successfully completed the course
  • My exam met expectations
  • I was one of the most active students across many foundations courses.

Aftermath

I have already stressed how interesting the material is and how much I’ve learned from the course. It’s good. The course is by far the best testing course I’ve learned about by now.

If you’re looking to send your testers on a course, take this one.
Really.

There’s a few things I didn’t really like though, but I can see why they are the way they are and how they each have their own function.

  1. I disliked my experience of the online component.
    While I really like functioning in diverse teams, I absolutely disliked it in this format. Don’t get me wrong I really liked the people I met and got to know. Maybe someday, we’ll meet again. But generally, I felt it was a distraction. Every task is focused on the individual, with the option to give feedback on others.
    Most of this feedback is about questions. “Why did you say/do it like that?”. You ‘lose time’ explaining your words and ideas, rather than have an in depth discussion.
    Sure, this is how it works in the real world. But I get enough of that in the real world already, here I want to learn and learn in depth.
  2. I disliked the Exam format.
    To me, the exam is not a good representation of “did I make the course or not”. It serves two functions: One; it’s a learning opportunity. A way to further process what you’ve learned. Two; it’s a measure of how well you completed the course.
    I felt it focused way too much on precision reading and precision writing than on what you’ve understood from the course.
    It’s very academic. I understand why, but that doesn’t mean I have to like it.
    I would really like to know how many from my class got their ‘certification’. That way I could take a guess at how high I should hold it in esteem.

I have felt frustrated throughout most of the course, but I have learned a ton.

Thank you, Cem, Altom, Ru, Gabi, the other instructors and all my fellow students for your efforts and knowledge sharing. I imagine I wasn’t the easiest student, but I’m grateful for the chance of learning with you.