This article explains how the Toss QA Platform team maintains the quality of Toss and fulfills its promise to users. To ensure a stable release of the Toss app, where hundreds of changes are applied every week, the team utilizes self-developed tools such as Toss Doctor and Toss Checker to perform smoke testing and regression testing. In addition, we are maximizing test efficiency by developing AI-based tools such as PR Analyzer and tcgen, and beyond simply providing tools, we aim for substantial quality responsibility and are setting Toss quality standards.

1. Toss, reborn every week 🔄

Toss is reborn as a new version every week, and on average, 300 to 400 code changes occur in one release. To ensure that users can safely use the Toss app despite these many changes, quality is a value that must be maintained by Toss and a promise to users**. If there is a problem with the app at an important moment, such as an urgent transfer, loan, or payment, the user's day can be completely disrupted.

The Toss QA Platform team is this 'someone' who stops every week before the app reaches the hands of users and asks the question, "Is this really okay to release?" Although they have tested numerous products this year, this is the first time they have told the team's story in writing. In the future, the team plans to share the path they have taken and solve the tasks that need to be solved one by one! 🚀

2. Thorough verification process before sending out the toss every week 🔎

The most important role of the Toss QA Platform team is to be responsible for the release of the Toss app. Before a new version containing hundreds of changes is released every week, we carefully monitor its quality through the following process:

• Smoke Test: When a new Release Candidate is posted, use Toss Doctor to quickly check that core functions are working properly**, from login to withdrawal. At the same time, we analyze changes with **PR (Pull Request) Analyzer** and conduct tests focusing on the scope of impact.
• Regression Test: Next, perform the Regression Test with Toss Checker. This is the process of inspecting the overall service to ensure that this week's changes have not damaged features that were previously operating stably.

The team's role doesn't end even after the app is released. Continuously monitor crash indicators to determine whether to respond quickly to issues as they arise or resolve them in the next deployment. Because a hotfix requires another update from the user, the team is said to ask 'is it really necessary now?' rather than 'quickly' resolve it. We don't forget to track crashes and hotfixes using our custom dashboard to prevent the same problems from recurring.

In addition to release work, the team is active wherever quality is required. It plays a role in raising the overall quality of Toss by providing direction to teams starting QA for the first time, guaranteeing the quality of in-house tools, and helping design QA processes.

The ultimate goal is that "anyone should be able to easily create test cases and test them quickly and accurately"! We believe that this is the key to protecting the quality of the Toss app.

3. Beyond simple testing, setting the standard for toss quality 💡

The slogan of the QA Platform team is "Going beyond simple testing, setting the standard for toss quality." Why is this standard important? The Toss team releases new features every week, and this is because the standards for judging quality should not be different for each person. No matter who sees it and when, it must be verified by the same standards. So the team defined the standards as follows:

1. Reliable Releases, Every Time: It is not difficult to release well once, but delivering the same level of trust to users every week is the real skill.
2. Pointing out defects sharply (Ensure Test Quality): Quality is more important than quantity in testing. The key is to accurately identify defects that can actually lead to serious accidents. That's why they say they focus on "what did they miss?"
3. Assuring Quality Efficiently: It is impossible to keep up with the changes that pour in every week by repeating things manually. So, I think 'not getting tired' is as important a standard as 'doing well'.

Complying with these three things every week is what the Toss QA Platform team calls the Toss Quality Standard.

This year, we decided to go one step further and expand the scope in which the test operates on its own. This does not mean that we will leave everything to AI, but the plan is to leave judgments that can be made to AI and focus only on where humans really need to focus.

4. Our own playground, Tossion 🛠️

As it became difficult to handle the increasing amount of testing workload every week by human hands, the team wanted to actively utilize AI technology for testing. However, the tools available on the market had many limitations. This is because it was difficult to fit Toss' fast distribution speed and work style.

So the team created our own platform, Tossion! ✨ Torsion started with the idea of ​​creating 'our own space' where QA team members could work more effectively and freely experiment with their ideals for quality.

Torsion started out as a replacement for TestRail, an existing commercial tool. The entire process, from test case writing to execution and result recording, was moved to Tortion. In this process, several bots were integrated into one bot called Toss Butler, creating a new bot to fit the team's way of working. After moving to Torsion, I started to see more things I needed.

• PR Analyzer (PRCheck): Since it is impossible to look at all the numerous PRs that pour in every week, we have added a PR analyzer that tells you to "focus on this." It is now possible to systematically check the bug risk and testing priority, including what and why the developer changed, what parts it affects, and what needs to be tested.
• tcgen: When large-scale changes or new features are added, the need to quickly write high-quality test cases has increased. So, we created tcgen, which collects PRDs, design documents, and other contextual information and automatically generates test cases. This allowed me to get a solid draft and focus on reviewing it instead of writing it from scratch by hand.
• Automated testing platform integration: With the desire to see manual and automated test results in one place, we integrated the automated testing platform into Torsion. Now you can see both results side by side on the same test performance page.
• Crash Trend Dashboard: We created a crash trend dashboard to identify crashes that occur not simply in terms of the number of cases, but also in terms of indicators and trends that fit the toss.
• Hotfix Dashboard: We have also created a hotfix dashboard that focuses on controlling hotfixes efficiently, classifying causes, and establishing measures to prevent recurrence. All of these tools are the result of one will: "Let's efficiently solve the changes that come every week." Each tool has a special story, and they plan to introduce them one by one in detail in the future in this series. I'm looking forward to it! 😊

5. The real feelings we missed 😮

The team hypothesized that creating and delivering test cases quickly would make testing much easier for people. I often heard voices from the field asking, "Is there an easier way to write test cases?", so I thought that many people would like it if this inconvenience was resolved.

So we created tcgen and released it, but contrary to expectations, people's interest was not as strong. There were clear voices saying it was necessary, so why not use it? 🤔

After thinking for a while, the team realized an important fact.

"What people were secretly hoping for was not 'making testing easy.' It was for someone to do the testing quickly and accurately for them, and take responsibility for the quality as well. That was their true intention."

The team thought it was lightening the burden, but it ended up giving users new homework. Just because you gave a good shovel didn't mean that everyone wanted to dig their own hole. This realization completely changed the direction of the QA Platform team for the second half of the year.

In the end, the team decided to handle the testing. We decided to move forward in a direction where we can achieve 10 times the efficiency and take responsibility for quality until the end** without having to hire people every time. It is a difficult road, but we are moving forward believing that this is the right thing to do.

6. Still, concerns continue 💭

Thanks to AI, many problems have been solved, but the team's concerns still continue. Just as a hypothesis is always half right and half wrong, this was the case with tcgen mentioned earlier. Moreover, AI technology is advancing so rapidly that yesterday's best practices may become outdated today. Amid such rapid change, we recognize that the moment QA insists on familiar methods, the tools created to protect quality can actually slow down the pace of change.

So the team decided not to settle for 'made it'. Once, we created API Labs to make API testing easier, but when we decided that the direction was not right, we boldly canceled it after 8 hours. I think that even if you put a lot of effort into something, you should throw it away if you think it isn't good enough and be ready to change direction at any time.

This flexible way of thinking is also reflected in system design. The tools created by the team, such as Torsion, Toss Doctor, and Toss Checker, are not a finished product, but were created with the premise that 'someday it will change' so that they can be replaced without hesitation if a better method exists.

AI solves a variety of problems, but it does not tell us 'what quality is' or 'what must be maintained until the end'**. AI can create tools faster, but it is still the QA Platform team's job to set the direction and re-establish standards so that the tools can continue to be useful.

7. Our story begins ✍️

So far, we have briefly introduced who the Toss QA Platform team is, what tools they have created, and what attempts and learnings they have experienced. From now on, they plan to unravel all these stories one by one in more depth.

Toss will be reborn again next week without fail. The screen will change, and hundreds of changes will pile up again. In front of that, the QA Platform team will stop and ask again, like it did this week.

"Is it really okay to send this out?"

Someone tests Toss, and the Toss QA Platform team is a special team that creates the testing method. I'm really looking forward to their story going forward! 😊