Tree of Savior Forum

A long QA story and a request for all founders

General Discussion
#1

tl;dr: 9 times out of 10 a game-breaking bug isn't fixed but is merely prevented. There is a difference. Please hit ToS as hard as you can going in, especially concerning forced frames and quest pop-ups, because chances are it hasn't been fixed but simply sidestepped.

I used to work in a sweatshop QA and this is a story from that time. There is a lesson here… and it applies to ToS, as well as any game. Specifics must be omitted but you’ll get the gist.

I had a project that was basically an annual title. Our company integrated their own built-in messenger services for ease-of-access for our users during online play.

It was a known legacy bug that the messenger itself, migrated from project to project, didn’t handle special characters well. During the start of every alpha we would see a full-blown keyboard system with everything you could ask for. Special stars, tildes, letters with all kinds of accent symbols, you name it.

By the end of every beta we would be left with a skeleton of that keyboard, with nearly every special character stripped from its place.

This was because spamming a chat message with nothing but special characters would crash the lobby. The next build this would be “fixed” but you would find that sending a private message with all special characters would still crash the recipient. Then that would be fixed. Except if you were in-game and received the message then you would crash. The build after, when everything seems fixed, you would find that filling the chat with anything you want but then adding a special character at the end of the message would still crash.

The list goes on forever. The fix? Remove all special characters from the keyboard. There. You can’t type them. Problem solved.

Fast-forward to early beta. Oh… if you plug in a USB keyboard you can still access some of those characters that crash the client. Okay, they “fix” that next. Starting to see a trend?

The issue was that special characters could lead to crashes but rather than actually fix the code relating to why the crash occurs, the simpler means was to just prevent the user from accessing them in the first place. If you find all possible ways the user can cause the bug then, in effect, it’s fixed… right?

Wrong. There will almost always be a way to cause that same issue if you really sit down and put your head to it.

One of the projects I was on… lol… it was the very last beta build. We thought we had it. We passed and cleared all of our required paperwork to sign off on the title for Microsoft and Sony. Half the studio was celebrating with pizza and tournaments to see who the best players were in-house. Until I found that if you entered in a dollar sign, backspaced to delete the dollar sign, and then entered a certain sequence of characters that… wait, what? It called up strings from the game? Really?

Yeah, the chat could still treat the $ as listing the associated string, in chat, with often humorous results. I stumbled upon it when clearing my message. If I could then someone else could too I imagined.

Sometimes it would list a place, sometimes it would list an item… people got quiet. The entire studio started messing around with the strings in-chat but nothing too bad happened. It was a “C” issue, not severe, not legal, not a violation, nothing. People gave up up on it, wiped the sweat from their brows, and continued the celebration.

Then I was able to squeeze out a standards violation to make it more likely to be fixed because one of the strings was tied to a Microsoft branding that, when referenced on a Sony platform, was a big no-no.

It was marked Known Shippable, meaning, it was never likely to be found because of the severe steps required. Specifically because even if someone found the string bug by mistake later on it wouldn’t really be likely at all that they would know how to reproduce the issue much-less know the exact string reference for the standards violation, or even know it was a violation in the first place.

I pressured it though. So did the studio. Over ten hours later… guess what? One of the strings… had special characters.

Boom.

With that long-winded and roundabout way to finally get special characters back into a message, it was all over. We could crash players in-game, we could crash players sitting in a lobby, hell, we could crash every single player in every single lobby. Regardless of how nearly impossible it would be for a user to find, the bug existed, and having the entire studio know of that existence increased the likelihood of it being leaked. Even one single reproduction in the real world would have been disastrous.

And I guarantee you that too this day that issue wasn’t actually fixed. All they did is stop the user from being able to display strings in-game and shipped the title.

But, the root of the problem, the special characters, was left to haunt the code forever more.

OoooOOoOOOOOoooooo. Spooky.

So, happy fun-time story yeah? The lesson is that bugs very rarely actually get properly addressed. They just get covered up and become harder to reach. I found it is always very important to treat every game like that since my stay at that company.

Our time in iCBT2 was riddled with issues that could stem into even more severe issues. We played around with font sizes and inserting graphics into chat. Some users found how to insert non-github graphics into chat even.

What happens if you push that image as a png with js injection into chat? What happens if, instead of just inserting quest dialog, we are able to complete quests from chat in specific instances?

I ask that all founders hit those iCBT2 frames as hard as they can. I hear this issue has been addressed for KToS… but… I’m not buying it. If a user can bring up even one single window when they shouldn’t normally be able to then the issue will not have been fixed but simply walled-over. i.e. the game-breaking instances were prevented from being brought up but the underlying issue of being able to bring up windows in the first place would still exist. If a user is able to pull anything directly from code (like github files etc) then there is a big issue there just waiting to happen.

A scratch can kill you if it gets infected and is left untreated.

7 Likes
#2

Yes. I will be hitting things REALLY hard and voicing any bugs/glitches I come across. Starting right at midnight (luckily for me my sleep schedule is perfect for the EA)

I didn’t read the entire thing; since it’s 4am here but I read the bolded part and parts of it. Overall: I agree. We need to hit this game as hard as we can and expose the bugs to be fixed. Not bypassed.

1 Like
#3

To be honest, the sheer number of bugs that seem to be cropping up with this game are a red flag for me… So I’m extremely cautious at this point… imc either needs to hire more skilled programmers or they need to take the time to get things like unit tests, better version control, and more detailed function/object class specifications working…

I mean they open sourced translation, why don’t they try open sourcing parts of the game code so the community can work on improving it. If they open sourced parts (especially with respect to the chat system or user inputs that would need to be sanitized) of the game they felt comfortable with sharing maybe that would help them squash bugs more efficiently without necessarily adding too much overhead on their end.

Maybe someone should go post “open sourcing parts of the game code” as a suggestion in that sub-forum. I would personally, certainly be happy to dedicate some weekend time to help optimize code, especially for some TP or title rewards haha.

#4

lol then you’re leagues beyond me. I took several courses in programming but sadly most of it went over my head. The logic of it all persists, though, even if the language didn’t quite sink in. So kudos to you on that one lol

I doubt they would share their code though for fear of reverse engineering.

I could be mistaken though xD And hmm hopefully this time around they offer at least some form of “bug-finder” item. I mean the title has cleared (I’m sure) countless in-house tests, two international closed betas, X amount of Korean closed betas (perhaps even a couple for a third-party QA team exclusively), and we’re going to be only a couple builds behind the KToS open beta? If there is anything worth reporting it should definitely have some reward this time around I would hope, considering this is something we’ve also payed to play in this next iteration. I mean… otherwise, wouldn’t it just be “$49.99 to help us find bugs” lol

#5

And yet it seems even in the current Korean build of the game:

It’ll get better sure, but I think that the game is still probably a bit too rough in some places to be fully released. Maybe that’s why imc wanted to have the true F2P launch in another month (or why they originally planned it to be in 3 months)…

#6

Unfortunately I don’t have a coding / hacking background… I don’t know how to intentionally reproduce bugs / break the game.

#7

It’s actually easier than you’d think.

For instance… go to any vendor in the game and tell yourself “There IS a way to get infinite silver from NPC shops, I just have to find it.” You can even go in with the mindset that someone else has confirmed it exists like maybe they posted a video of their trillions of silver and them buying everything on the Auction House that they want. All you have to do is reproduce it.

The key is to go in knowing that everything is broken. It’s just up to you to find that sweet spot to make it break.

So you go in with an objective. Tell yourself there is a way to get infinite silver from a vendor… how would you do it? Well, maybe you can somehow sell more of an item than you have. If you sell 2, but only really have one, then maybe you could buy them back at the cost of 1 and make a profit every time. Maybe that would even let you duplicate items. This actually led to me finding out that you can buy fractions of an item, or even 0 of an item, and have it count toward your achievement score. It didn’t actually cost any silver at all because no matter what the vendor said the fee would be, my silver wasn’t increased or decreased, but I did manage to trick my achievement log into thinking I had actually acquired the item. This also led to being able to purchase items that would bug out your inventory, as buying a few fractions of an item then buying the item like normal would result in it taking up to 20+ spaces in your inventory bag, its slot and a ton of empty placeholder slots.

Those are trivial though. The achievement one did lead to premature padding of your scores and ranking, and the item taking up 20 slots didn’t do a lot either. For me at least. If the bug still happens, you just repeat the steps. “I know that the item creating 20 blank spaces in your inventory leads to a crash. I just have to find out how.” Can you make someone else crash by listing it on the marketplace? Does it make the market fail when it tries to display the item? In a 1:1 trade what happens? What happens in a 1:1 trade if you fill up all your trade slots and put the item in with 20 blank slots? What happens if you trade someone 5 of these items?

Just keep thinking of ways that you can find that break.

I spent millions on anvils last beta. I know there is a way to trick the system into being successful every single time. Maybe it involves using certain inventory slot combinations, because maybe the code doesn’t handle it as it should. I’ll tell you though, I tried a lot of things. I hit it to specific server times. I tried with 24+ hour gaps thinking maybe there is a hidden value that punishes successive attempts. I had skeletons hit it in sync with my hits because minion hits didn’t register. I tried only using attribute damage from swell body to ding the anvil. I even tried using certain weapons and sub weapons to hit the anvil. Oh, and rest assured, I tried the upgrade processes on those bugged out extra-space items too lol I even tried to sabotage other player’s anvil attempts thinking maybe there were things I could do to influence their success rates for better or for worse.

The point is though, I didn’t have any results, but I still know that it is broken somehow, I just have to find out how. And with enough time I will or someone else will lol

Maybe even you.

#8

Maybe if they placed a bounty on discovered bugs, people would be motivated to test. I think they did this already on the previous beta.

#9

So you worked at place where everyone was so **** at coding that they removed features because they couldnt fix the issues related to them.
Your conclusion: Everyone sucks at coding, be afraid.

Its sweeping bias wrapped up in the concept of a blanket statement.

#10

Eh… my view on his story was that the dev team essentially took a white list strategy to input sanitation… white lists on the whole are generally more secure since you can never be sure you’ve covered every possible blacklist input (the unknown unknowns).

https://en.wikipedia.org/wiki/Secure_input_and_output_handling#Whitelists_and_blacklists

#11

lol quite possible, I can’t say either way since it definitely wasn’t my area of expertise. In fact the reason I was there was because I was still interested in the industry even if I decided that coding was beyond me.

Hmm. Well, they weren’t bad at all, but they were definitely strapped for time. The studio worked on no less than…well, that year, I think they had 8 titles in-house, with as many as 4-5 running parallel at any given time.

So, true, it is probably a practice they adopted that not every studio is by default bound to. However, depending on how deeply the issue is rooted, and how far along the project is, the more likely it is that any studio would likely try to code around the issue lest they break more than they fix.

Since we’re so far along (as mentioned, at minimum 5-6 public beta cycles, not including in-house exclusives or alphas) I doubt that any severe issues would be easily fixed. Likely there are too many dependencies throughout the code and making a drastic change wouldn’t be their first choice.

#12

Good read. I’ll be sure to report any bugs I find. Unless it’s just an insect type enemy. Then I’ll probably fix it with a dozen arrows. :laughing:

1 Like