Continuous Delivery using Crowdsourced Testing

On Wednesday 14. March 2018 I presented my talk “Continuous Delivery using Crowdsourced Testing” at the Swiss Testing Day conference. It was a pleasure to talk about the topic which occuped lots of my time and thoughts in the last one and a half years. Even if presenting on the main stage at the Samsung Hall made me just a bit nervous at the beginning, I really enjoyed it.

I’d like to underscore the most important learnings I’ve made working intensively with crowdsourced testers and incorporating the crowdsourced manual test cycles into our overall agile testing strategy, while working in a fast-paced DevOps environment. 

1.

It is important to understand what continuous testing is and where it fits into the DevOps culture and mindset.

Continuous testing is a process of executing many different types of tests — both manual and automated – continually throughout the delivery process
to obtain immediate feedback
on the business risks
associated with a software release candidate.

  • Definition adapted from Jez Humble. I encourage you to read his Continuous Delivery book.

2.

Testing actually happens on every stage of the DevOps toolchain. Note that this article from Dan Ashby is from October 2016 but lots of the people still don’t know about it and how true it is.

3.

In our efforts to continuously deliver online storage solutions for our private customers we never forget the manual validations. Automated tests run continuously on our development, integration and test environment but how do you scale and get fast feedback on your UI (web, mobile apps) or usability tests? Also, you can’t automate all your customers flows.

Continuous Deployment Pipeline – note the manual validations phase

 4.

Ideally you should, but usually you don’t have an experienced tester in every cross-functional development team. Also, lots of the time it can happen that the team can’t finish all manual or automation tasks before the sprint ends (here I’m not talking about the other causes for sprint-spillover). And even if the team tested all sprint artefacts, how much time would they have to run regression, exploratory, usability or user beta tests?

To compensate the activities for which you’re not skilled or just don’t have enough people to get dirty (which IS part of the game) you can hire a testing service provider who maintains the crowd: community of experienced testers and IT specialists. Having that complementary testing service to internal (testing) teams with a focus on the end-user experience, will certainly help you reach your sprint goals and increase the quality of your products.

5.

Main advantages of crowdsourced testing

  • Resources flexibility
  • Recources scalability
  • Fixed price contract normally includes the hours of CT company’s internal stuff (project manager, technical leads) as well as any possible compensations to the testers for the issues they find.
  • Provides user point of view needed for testing user-centric apps
  • Crowd is not confirmation biased
  • User surveys can be included in every test cycle
  • You can also test mocks and wireframes
  • Crowd testers are rated based on their performance on past projects -> you get an access to a large pool of good testers

6.

Even if the crowd test execution cycle and the whole process might look simple there are a lot of challenges to solve on the way.

  • depending on the product your developers might get overwhelmed with the number of found issues. Some of them will start rejecting them or the process in general.
  • lots of the results found might lack the significance or impact, meaning they get approved as somewhat valuable issues. Remember we want to find those exceptionaly valuable issues that involve business risks.
  • internal coordination & communication time needed can develop into another bottleneck, especially if you’re the only one capable or willing to do those tasks. Already after I started, I found myself spending the half of my working time “only” on preparing next cycles and analysing the results of the current or past ones. That didn’t leave me with enough time for other strategic activities or building up on my test-automation skills.
  • even if the crowd is huge you still paid for some particular number of testing hours per month and having only good testers claiming their slots on a first come first served basis is pure luck. Of course you can invite the names from your “top 20” list but nobody guarantees you they will  really claim their slot and test
  • besides having the good testers it is very important to have a great testing team lead on the CT company side. Imagine having and then loosing a person who’s a brain of your test execution cycles who knows and understands your products, the known issues, remembers what was found in the last weeks and why. When such a valuable resource is not available anymore, it will take lots of luck and some weeks to onboard the other one (or you will have even more work).

7.

There are certain strategies to solve those problems but you need to know where to start and that is normally where it hurts the most.

  • Stick to (your) timeboxed time and delegate all different preparation or follow-up tasks to your team -> be persistent in that and don’t give up
  • Educate the team about the importance of “three amigos” principle, meaning that not only Product Owner is syncing with you on the issues found or on the tests planed to be run or updated, but the developers as well
  • Start buddying and pairing with those developers who value the CT process and don’t forget to appreciate and thank them for their help and support
  • Make sure that the issues found from crowd test cycles get into the next bug refinement
  • Track improvements
  • Continuously adjust your strategy and process and make it flexible
  • Never forget to star and reward good testers and even send them a personal thank you note
  • Never stop educating your team
  • Never stop learning 🙂
    • Also learn from other testers findings in terms of improving your testing skills and mind-maps

8.

Further things to keep in mind:

  • preparation is the key ingredient to your success and good input creates the good output
  • information on which the particular test cycle is based need to be clear and precise
    • Test scope / products to test
    • Software version / environment
    • Test plan (if you want them to run regression based on test scripts)
    • Known issues list
    • Internal release notes
    • Sprint review video demo & presentation slides if any
  • be clear about how you’re using the CT report dashboard with a results summary per risk, devices, browsers, technical or functional etc.
  • concentrate on testing with the devices your real users use the most and get those numbers from the analytics
  • beware the internal costs needed for coordinating the whole process 

9.

Summary:

  • Know your CT goal & purpose and aim for the “highest” added value:
    • What
    • How
    • When do you want to crowdsource your tests (in our project it’s normally towards the end of the sprint, every 2nd week)
  • CT won’t solve all your testing, QA or software development problems, will though help with functional or user tests
  • test internally as well 
  • every cycle should include explorative tests – the results are much better than with regression test scripts
  • setup the two-way integration with your JIRA
  • insist on keeping the good testers in your next cycle and try to attract them by giving new products and features to test, otherwise they might get bored or will stop testing your current product(s) since they know they won’t find any more valuable issues (less money earned for them)

10.

We are using Applause manual testing services (formerly: utest) and have been satisfied especially with the professionality of their internal stuff who really knows about testing. Check out their how-to pages to understand in more detail how bug verification, two-way JIRA integration and test management  process work. If you have any questions where I could certainly help with my experience and tips, don’t hesitate to contact me.

 

I’ll write about the conference in the next post. Stay tuned. 🙂 

Advertisements

Quest for Quality 2017 conference Dublin

On 4 and 5. October I had the honor to speak and take part at the Quest for Quality conference in Dublin, Ireland, organized and sponsored by Comtrade Digital Services. It was the software testing conference with a focus on the challenges that Quality Assurance in IT industry faces within the digital transformation. Quest for Quality began its journey in 2016 with a very successful 2-day event in Ljubljana, Slovenia and what a great event this one was! It offered a great mix of excellent content, superb location, nice stuff and attendees and perfect network posibilities.

For me the conference began already on a day before since I wanted to be there on time and also I was invited to the speakers dinner on the Tuesday evening. Through the course of Apéro and dinner I had a chance to meet the organizers and already make friends with them and my fellow speakers. We tackled different topics including our talks but also learning and coaching about agile testing mindset. Also, tips and tricks when organizing local meetups were on the agenda.

1st day

On the next day the conference started with the opening notes by Viktor Kovačević, Director of Comtrade Digital Services and Nikola Šopar, Head of Editorial Board, followed by 3 keynotes. I was already amazed when I realized there were 3 keynotes on the first morning and what a superb start that was!

The first keynote by Gar Mac Criosta was about Quality in an Age of Uncertainty. 

Digital technologies have changed the way we work and live and that change is accelerating. Platforms and the platform economy are creating new ways of delivering amazing experiences and crazy new connections between customers and providers of services.

Gar Mac is a digital architect, instructor and a founder of Business Model Adventures.  If you check his company’s website you will find some nice pointers toward Happifesto but also his inspiring happy startup e-book.

In the second keynote Andreas Grabner from Dynatrace talked about How DevOps is done by Top Performers.

DevOps is one of the most abused and overrated marketing terms in the last years! That’s not an alternative fact! It’s just Andi’s opinion! YET – it is a very real thing that allowed many software companies to transform the way they think about software engineering. The downside with DevOps in 2017 – based on the recent State of DevOps Report – is that DevOps transformations are most often driven by business which means “Faster to Market” but not “Faster Quality to Market”.

I couldn’t agree more with everything Andreas explained and some of the key getaways are:

  • Being responsible for the features end to end
  • Idea and example taken from one of the top performers: boot camp where they challenge engineers to get out of their comfort zone and try out and learn the things they don’t like
  • Quality heat map including the baseline test metrics for e.g. 10 most important tests.

The third keynote about Business Value of Testing was by Dejan Ćušić, Business Director at Comtrade Digital Services in Dublin. Again the topic lots of us want to master. Dejan delivered very good tips and examples.

While it may seem obvious to QA managers and testers, business is very often questioning (rightly) business value of automated testing.

During and after the lunch I had to take care of some work stuff so the first afternoon talk I visited was one by Maik Nogens about How the XING Team Created a Successful Content Platform from Scratch. This talk can serve as a reference for lots of other development teams.

An overview of how testers are involved in XING, how a strong tester community was build and how we handle the balance of continuous delivery in web versus the constraints in submitting to App Stores. The talk also gives a glimpse into a typical iteration for a tester in an agile development team and what (maybe) non-typical tasks are done.

Brendan O Reilly explained Why Culture eats DevOps for Breakfast. I’ve heard him talking about Pathological, Bureaucratic and Generous culture and liked all his views but soon it was already time for my talk. 🙂

I was talking about the topic that kept me very busy in the last year, Continuous Delivery Using Crowdsourced Testing. Based on my practical experiences with adjusting our agile testing strategy to include crowdsourced testers for our manual functional (usability, beta, new feature and regression) tests, scaling those resources to enable us getting to know our end-customers even better, using analytics to decide what devices, browsers and OS to test in more or less extent, I was very happy to see lots of attendees being very interested to hear my talk and ask me lots of valuable questions during Q & A and in the next two days.

I have to admit that after my talk and since it was already 4 PM I was tired but happy enough to pay myself a real break. Also some more work stuff waited for me and therefore I couldn’t really follow on the AI talk and panel.

The first conference day ended with a fabulous networking event in the Marker Brasserie. Fed by delicious food and inspired by the cool disco DJ sounds one could dance or mingle with the colleagues and exchange views on different subjects.

2nd day

The second day started with the Finn Lorbeer’s (Thoughtworks) keynote about Building a High Quality Product. 

Finn argues that this is only one part of the story: we do not only need to test earlier in the software development process: we also have to apply other methods than testing to ensure the fast and robust delivery of an overall high quality product. Additionally, we need to apply those methods in the different context of our work. This includes understanding the business value as much as the system architecture of the product.

Finn said:

  • Testing does not build quality. QA is just like a light showing what is there on the way.
  • Don’t waste a mistake, learn from it.
  • You may not have all the right skills among your project team. In that case, advocate for swapping people in order to have the right skills in your team.
  • Having fun is one of the most important things.

The following keynote was by Ingo Philipp from Tricentis which was so far my favorite talk of the whole conference. While talking about how to Rediscover Exploratory Testing, Ingo was funny but also delivered high value content and lots of tips how to properly execute exploratory tests.

The talks that rounded up my second day were, among others:

The closing keynote was delivered by a beautiful lady and communication expert Gina London. Here are some key takeaways from her energizing and a bit provocative talk:

  • We are never NOT communicating
  • Don’t be distracted when you are communicating
  • First impressions count
  • Are you doing something deliberately or by default?
  • Warm embrace, gracious handshake
  • AIM “method” meaning:
    • Audience analyze before you communicate. It’s about them, not you.
    • Intent what do you want them to do ? goal ? Action ?
    • Message should be memorable, brief, emotional, databacked, craft with care
  • what you do is up to you !
  • ‎introverts understand system and structure and can skill up better
  • communication should be ‎vibrant, compelling
  • written communication is also very important, but don’t put too much in an email.
  • when you read an email, highlight the most important parts
  • Finding a balance between so much data and communication is very important -> layers
  • Fear the repercusions when not practicing public speaking, and don’t fear the public speaking itself.
  • It is very passive aggressive to write when you sit accross the hall. 🙂

THANKS

I’d like to give my HUGE thanks to the organizers of this remarkable conference, all speakers and attendees and especially to Viktor, Aleksandar, Nikola, Dejan and other Comtrade colleagues who amazed me with their knowledge and insights and to my new friends and colleagues Evelina Ema, Amela, Darko etc etc. 🙂 Thanks Lina and Neven for having such a nice stroll through the city with me. It was so much fun with all of you and I’m looking forward to see you all soon.

Photo Album

Here’s my Quest for Quality 2017 Dublin photo album. 🙂

15th Agile Testing Meetup Zurich: The Magic of Testing & DevOps

End of August there was another agile testing meetup, this time on the topic of testing in DevOps environment. It was a very successful meetup with around 18 participants who underscored  they would like to have a sequel on the same topic soon.

Prior to the meetup and actually just on time the package with the EuroStar 2017 TeamStar material including the most delicious chocolate arrived, all of that organized by Ronan Healy.

Thanks a lot Ronan and EuroStar. 🙂

Meetup schedule

17:30 – 18:00 – Welcome and short introduction / get to know each other

18:00 – 18:30 – Presentation by Maja about her experiences with DevOps and Testing

18:30 – 19:00 – Presentation by Simon Berner, House of Test

19:00 – 20:00 – Open discussion

After picking up the attendees and showing them the directions to our meeting room we first had a short introduction round. It was nice to see old and new faces in the audience.

Talks

The first talk was by myself. Often testing can be viewed as a barrier to implementing Continuous Delivery and DevOps, but it shouldn’t be. I explained what DevOps is and how Dev and Ops are intervened with QA. Since continuous testing means continually running automatic and manual tests throughout the delivery process, in order to successfully implement it and be able to run your manual tests in a short period time with scaled human resources I explained how we are successfully using crowdsourced testing as part of the testing strategy.

Second talk by Simon Berner was about Testing in a pipeline. Simon showed detailed examples of their continuous deployment environments and how they are covering automated Unit and UI E2E tests.

Main takeaways:

# crowdsourced testing can really help you meet variable demands and improve time to market

# building a state-of-the-art pipeline is very important for testing as well

# read the new book from Katrina Clokie about Testing in DevOps. 🙂

# timebox the sessions better for the next time so we have enough time for chat

 

Attendees feedback:

# An exciting overview of your setup and your efforts in the DevOps context. Unfortunately, a little time to chat.

 

//platform.twitter.com/widgets.js

 

//platform.twitter.com/widgets.js

 

//platform.twitter.com/widgets.js

To be continued 🙂

In the next months I’ll organize the sequel to this topic and am very much looking to meet my agile testing fellows again.

 

DevOps Days Zurich 2017

DevOps Days Zurich 2017 were held on 2nd and 3rd May in Winterthur and it was a superb event with lots of interesting talks and cool possibilities for networking and exchanging ideas and knowledge with very skilled and experienced DevOps folks.

Already in the first talk by HumanOps champion Hannah Foxwell about “Tech is easy. Humans are hard.” one could learn about the very human aspects of DevOps and topics like CALMS and four pillars of effective DevOps.

CALMS

CALMS is a conceptual framework for the integration of development and operations (DevOps) groups, functions and systems within an organization; the acronym stands for:

Culture
Automation
Lean
Measurement
Sharing

Next talks were about distributed systems, organizational aspects of DevOps, DevOps Security, etc. but here I don’t want to retell you the talks but to emphasize some tips and key learnings from the conference:

There were lots of questions and tips about learning and know how transfer

  • In Hannah’s company, every 6th week the whole technology team (I’d say: why not the whole project team) spends 1 week on self-directed learning. So, no project work, just learning and advancing in your master skills.
  • Lots of attendees striked out the need for pair programming, code review, configuration as a code, investing time in writing proper automated tests (checks) – actually everything one should already know but sadly lots of companies and projects miss to fully implement.
  • Sharing knowledge has to be made an organizational process
  • Everybody (!) should do a release at least once
  • Learn how to motivate developers to review each other’s code even if they are not proficient with another technology (web, backend, mobile app frameworks). Surely after some time they’ll be able to do that and the knowledge transfer (in some degree) will also be assured.
  • When reviewing code the most important part is to understand how and why the code was written in that particular manner.

Key take-aways

  • Every DevOps team needs a common vision and goals
  • Focus on inspiring people and not on tools and technology
  • Mindset is the most important thing
  • Continuous improvement is continuous – the journey never ends

Thanks to the organizers and sponsors for making this great event happen and see you soon at the next DevOps Zurich Meetup.

All video sessions and slides will be available soon (I suppose at the DevOps Days website).

How do you really wear all hats as an agile tester: 14th agile testing meetup

 

After the summer break we met again to discuss how do you really wear all hats as an agile tester.

We discussed the topic from two different perspectives.

There is a lateral thinking concept of Six Thinking Hats designed by Edward de Bono in 1985, which can be used not only in project management but in (agile) testing as well.

These 6 color-coded Hats include:

  • Managerial Blue. What is the goal? What is the subject? Why are we here?
  • Informational White. What are the facts? What are our resources?
  • Emotional Red. What does my gut say about this project?
  • Discerning Black. Where are the danger areas? What can go wrong?
  • Optimistic Yellow. Where are the opportunities? What are the benefits?
  • Creative Green. Can we make this better? Can we incorporate new ideas?

Source: Teststuff

Second perspective are all different roles (hats) an agile tester should fulfill (wear) when working in an agile context, like QA, Test Engineer, Test Manager, Software Engineer in Test, Test Master etc.

imag29351

After I shortly explained my point of view on these two perspectives the discussion flourished. We agreed that one should use / wear all hats but one more than another depending on his character, personality and context. It would be nice to have and form a team of different individuals who can complement each other and further grow and learn together.

imag29341

Thanks to all participants for coming and sharing their experiences. We’ll meet again in autumn to talk more about lessons learned.

Continuous Testing: discussed at 12th Agile Testing Meetup in January

In January we had another agile testing meetup, this time about continuous testing.

We discussed what continuous testing means and how does it fit into the Continuous Delivery process.

The definition for CT as “continuous monitoring with checks” was found correct by most of the group.

CT depends on a context

Tools:

  • Fitnesse
  • JBehave
  • Selenium

-> different learnings / limitations

-> develop a good test strategy

For testing the middleware it would be the best to develop your own tools.

The amount of the automated UI tests should be minimized, because:

  • they’re slow
  • expensive to maintain
  • dependent on a browser -> use REST to test API’s and business logic

For all those who would like to learn more continuous testing, I suggest some of the books already referenced on the Wikipedia page.

About James Bach – “How Accurate is It? Three Hours of Deep Testing”

Last night Ilari Aegerter and Ergon Informatik AG organized the Context Driven Testing Meetup with a special guest James Bach. Almost 70 people came so it was a great networking event as well. Ergon offered their stunning restaurant / meeting place and some snacks, soft drinks and  beer so it was a really nice atmosphere.

James presented what he learned and tested in a so called “deep testing” 3 hours session.  Shortly said, he tested a Raw visualization tool and showed us how he tested a “Parallel coordinates” chart.

What I liked and found interesting

  • I liked his approach with choosing a geometrical form (circle) to test visualization results, the way how he programmed the generation of the test data (even if they weren’t random nor exhaustive) and of course I liked his testing madness 🙂 and attention to detail while trying to find the bug
  • The best thing James showed was the conclusion including 3 criticisms:
    • he didn’t run any stress tests
    • the issue with the test data and it’s “wholeness”
    • “Parallel coordinates” chart / the functionality he tested, is too simple to be producing a bug, so he chose the wrong test for deep testing and spent too much time on it
      • Still, the learning and e.g. test data setup don’t have to be repeated next time
      • During his presentation I thought “but what about all other charts” 😀 and how he prioritized his test(s) but thought “this must be just a small simple experiment for the whole audience to understand it properly”
  • There was a nice discussion about if and how much his tests were accurate
  • What I liked the best was when he showed the graph of his “one week of test” and “seven minutes of test” sessions and how much of that time  he was actually testing and how much spending on test setup, talking to development, talking to management, having lunch and coffee and at the end what happened after the new delivery.

IMAG3490

Screen presentation photo “one week of testing”, Copyright James Bach

  • The citation about “different freedoms” there have to be in a team and how everybody has to have a different freedom in order to have all of them united together in a team goal was one of the best things I’ve heard in the last weeks. Marvelous James, thank you !!!
  • Pay attention to this blog about Snowflakes and Clods and the book “Tempo” by Venkatesh Rao. Again, thanks for the different view and inspiration.

What I didn’t like

  • Honestly, I didn’t like James’ attitude when he said that he doesn’t really know how Raw is working and that he couldn’t or even didn’t look for a documentation. I don’t find his strict opinion about “there is no proper documentation” true. Maybe there is a documentation but I’ve got an impression that he didn’t really wanted to have it.
  • Also saying “I didn’t talk to the developers” is in my opinion counterproductive and not how a tester should actually work with development.
    • So, guys, trust me, never forget talking to business and development !

All in all

A fabulous learning and inspiration hour for the beginning of just another one week of testing. Thanks James, Ilari and Ergon.