Balance testing ideas

[gruevy]

It doesn't appear to me that there's any really organized approach to balance testing. I asked DarkGod about it, and I got the impression that I was correct. I think the need for a more organized approach is pretty obvious at this point because it's pretty plain to everyone that some classes are too strong (cursed) and some are too weak (temporal warden). When people talk about balance in the IRC channel, I see a lot of words like "it seems like" and "feels". These aren't words based on numbers and data. Here's my recommendation on how to do some truly legit testing, in a reasonably fast manner, to see what classes and builds are viable under what situations.

I call it the balance arena. Come up with a better name if you like. The idea is that you create small, organized mazes with triggers at either end. These triggers would basically represent stairs. The mazes would be able to take many shapes - long corridors, random trees, etc. They are filled with varying numbers of creatures at, above, or below your level, as desired, and you fight them. Once you've killed them all, you hit the trigger at the other end, and they respawn. This continues until you die or stop. No loot drops and your character never gains experience (unless you want them to, of course). You would begin with a specific build, including stats and gear, and just gather data.

You count all the turns that happen, and measure things like the following:
-How much damage happens overall, and get a per turn damage number
-How much healing happens, what type it was, and how much happened per turn
-For example, how much healing was from infusions and how much was from skills.
-How often you activated abilities, counting sustains and activated separately
-How many enemies were killed with what skills
-How many enemies died with what debuffs on them
-How much faster enemies died with debuffs than without
-How many turns a character lasts, on average
-How many times a class dies to a certain boss, over 150 tries
-Freaking anything else you can think of

Here are some scenarios that I'm envisioning, just as examples of the kind of data you could gather.
-Rogues seem really weak to start out, so you could test various builds at level 3, and then drop Bill in, then kill Bill 50 times, or something, and see how you fare against other classes.
-Sun paladins seem weak after level 30, in my experience. Create a bunch of level 35 paladins and test them for a while until you see what builds are strong or weak, then buff or nerf accordingly.
-Test every class against 100 enemies at your level, then drop in Bill. You gain experience, and fight bill at that level. This helps you test the exp penalty vs. the benefits in the early game. Or, just test certain boss fights that appear late in the game with, say, 1000000 experience or something, given to level up with. You could see how a higher with %15 exp penalty fares vs a cornac with no penalty, due to the extra levels he'll have.
-People complain about skeleton mages. Test each class at level 5 against wave after wave of those, and see what succeeds and what doesn't.
-Test each class against random enemies 3 levels higher, then test them all against single types of enemy 3 levels higher, one wave at a time. See which classes fare better in crazy situations.

To sum up, just imagine, when some newb complains about Sun Paladins being underpowered, instead of going, "No they're not!", saying something like, "Well, they last on average 10% longer than half the classes, do 5% less damage, but defend against 10% more attacks and damage, at level 40. Also, I've run them through the end game fight 25 times in greens, and won 22 times."

This is just a rough outline of an idea. If anyone likes the concept, it could certainly be improved.

EDIT - One last thing. It's probably a good idea to make sure that your testers meet a minimum skill level, such as people who've won the game, to make sure that your data is accurate.

[Grey]

EDIT - One last thing. It's probably a good idea to make sure that your testers meet a minimum skill level, such as people who've won the game, to make sure that your data is accurate.

Well that reduces the potential size of any testing team quite significantly ;)

I'm not sure that such a complicated system is needed when people can just play the game and report instead (see the Class of the Week thread for example). Also classes shouldn't all be equal in difficulty, and quite frankly can't be anyway. Every class should be playable and winnable, and every class should face challenges too, but it can never be equal across the board. Some classes will inevitably be stronger in different areas, and some will ultimately be a struggle to play, and that's not a bad thing.

More important in my opinion is balancing of talents within classes. There shouldn't be talents that no one ever uses, and there should be more competition of choice between the good talents so not everyone chooses the same build every time. Talents should stay useful for a good portion of the game too. This requires in-depth analysis on a class by class basis.

[gruevy]

Firstly, vague notions about "everything shouldn't be the same difficulty" are doing more harm than good. It's true that some classes will, despite any attempts at balance, be easier, if only because how to succeed with them is more obvious. That's really sort of beside the point because I'm not actually talking about balancing them perfectly with each other; I'm talking about balancing each class with the game itself.

(EDIT - Let me clarify one thing here. It's more important that each class have a reasonable likelihood of surviving at each level, if built correctly, than that they all do the same damage or healing or whatever. Perfectly built characters should all have the same likelihood, if played correctly, of successfully clearing much of the content. One boss will be harder for one class than another, but I'm talking about taken as a whole.)

Besides, there needs to be a way to MEASURE how different they are, and how viable they are vs other classes. If no matter how you build em, Class X is weak at level 20, then that's a problem. Right now, no one really knows. Making one character that's successful or unsuccessful doesn't actually tell you about the class as it would be played by thousands of people. There are too many unknowns in any single player's assertions of balance anyway, like skill, build, eq, etc, to base balance on that.

But you bring up on big reason why something like this should happen: some skills are just too weak to be viable, and it's very hard to make a fact-based judgment on which ones, and how much they need to improve. If Skill X is crap UNLESS you build your character in a specific way, then that's not a problem. It's something for a player to figure out. However, if Skill X always sucks, then how would you measure it, esp against other skills that other classes have?

[marvalis]

The main problem with early game balance right now is that armor is a flat value while weapon bonus of skills is often a percentage bonus based on weapon damage.
Percentage bonus scales better but starts lower. Flat value start high but do not scale.

If we could somehow automatically collect data (level, damage done, monster hp, ...) and compare it with skill coding, then we could rebalanced things.

The normal (manual) way of doing this is simply starting a new game, and recording everything relevant for balance of that class.
You also might have to make excel tables to compare wanted values with projected skill values.

Some games use 'damage normalization' for skill balance. I am, by far, not an expert on this but the general principle is that you register all the important values (hp scaling, level scaling, ...) and you decided the rules for balance. For example, you determine a max/min relationship between level and damage of an item or skill. Then all the skills/items you create have to meet the requirements of this formula.

I think the best thing you can do right now, is make a place where we can register (document, record) values. Like for example a wiki page where we put down normal hp scaling rules, where we put data like max stats/level etc etc.

For example: Here is the data of stat scaling and level:

game/modules/tome/dialogs/LevelupStatsDialog.lua

Code: Select all

if self.actor:getStat(self.sel, nil, nil, true) >= self.actor.level * 1.4 + 20 then

Here is a table of player level, how many stat points he has gained, and what the maximum amount of any stat is for a given level. From this table you can calculate how much damage your skill would do based on that relevant stat.

Code: Select all

Level / Addition stat / max stat
1	3	22
2	6	23
3	9	25
4	12	26
5	15	27
6	18	29
7	21	30
8	24	32
9	27	33
10	30	34
11	33	36
12	36	37
13	39	39
14	42	40
15	45	41
16	48	43
17	51	44
18	54	46
19	57	47
20	60	48
21	63	50
22	66	51
23	69	53
24	72	54
25	75	55
26	78	57
27	81	58
28	84	60
29	87	60
30	90	60
31	93	60
32	96	
33	99	
34	102	
35	105	
36	108	
37	111	
38	114	
39	117	
40	120	
41	123	
42	126	
43	129	
44	132	
45	135	
46	138	
47	141	
48	144	
49	147	
50	150	

It means that for balancing purpose, players will max out any stat at level 28, and the second stat at:

Code: Select all

(60-statbase1+statmod1+statbase2+statmod2)/3 + 20

that is about level 31, and a third at +/- level 47.

More on topic: data is aggregated on the website, we can see how many winners they are, and that tells us a lot. Improving these statistics might learn us a lot. The challenge will be transforming the data in useful metrics. The we would have to propose metrics. In order to measure damage output, we would have to code the game so we can distinguish between combat and non-combat. Then we can calculate average damage during combat. etc.

[Canderel]

I like the idea, but it's too tedious. How about one just looks at the classes and their skill trees, and discuss a couple of things:

Per class (leave race out for now) have:

• Damage output (DPS in Warcraft, or Damage Per Turn in turn-based games) per level per class
• Damage resistance/Total life (also includes specific resists like precognition, resolve etc.)
• Disables available as well as how often are they available. (the cooldowns are essential)
• Crowd control abilities (Fireflash, inferno, scatter shot, whirlwind etc.) even beam spells
• Information gaining abilities (Track, Arcane Eye etc.)
• Evading monster abilities (invis, stealth) (note these don't gain you exp, but might let you life through those orc ambushes)
• Mobility abilities (rush, disengage, Teleport, Phase door, probability travel etc.
• Max class "options" (Total class skills * 5)
• Max generic "options" (Total generic skills * 5)
• Others stuff that I missed in the above mentioned list

If you can list these (per level) you'll be able to tell a lot of stuff. One interesting thing though is that at lvls 1 - 20 most classes have a *lot* of options, almost too much to choose from. At levels 20-40 you have a good "base" of skills and you are generally just increasing something you already have. And well, lvl 40-50 your options seems to deplete again, as you are maxxing out all of those you really want, and saving the rest to spend on the skills that will really make a big difference.

[marvalis]

More on topic: data is aggregated on the website, we can see how many winners they are, and that tells us a lot. Improving these statistics might learn us a lot. The challenge will be transforming the data in useful metrics. The we would have to propose metrics. In order to measure damage output, we would have to code the game so we can distinguish between combat and non-combat. Then we can calculate average damage during combat. etc.

Thinking further along this line, we could have the game calculate, for each level, the average damage the player has done (for every game). Then have the game store this one number in a variable for each level. Then we can gather that data to the profile server. Then we could somehow plot that data of several games (based on variance, average, median damage etc.). This could technically all be automated, and this together with other metrics (like average damage taken during battle) would give us an interesting perspective and measurement on balance.

[gruevy]

Canderel: one reason why it's hard to simply take a dps census of the classes, incl looking at movement, cc, etc, is that there is a lot of synergy that doesn't need to happen in wow. In WoW, you just do damage, period. Or you tank, or whatever. In this, you have to be able to do everything somehow. Having level 1 Rush, for example, makes your effective damage higher if you use it often, because of the smaller amount of time you spend running up to enemies, and it makes you live longer because you can get to archers and mages faster. I can't really think of a good way to measure that. That's why I recommend a system like this, where you let the best players create the data, and measure it that way.

[Canderel]

Yes, I wanted to say "theoretical DPS" but thought well, I mean only whilst in combat, and people should assume this.

So it's your absolute max DPS, and also an average DPS is also a good idea.

It's very hard... Archers (which are probably one of the easiest to calculate) would be Mastery + Steady Shot + <Bow> + Crit chance (divided by 3 for steady shot cooldown)

But:

Writing a lua script which creates a wall monster (100,000 HP, no regen), create a char (save it) let it attack using only 1 skill until dead. Then use the next skill, and recording it's max, min, ave, and turns it took to take it out would be possible.

One just need to set the level, stats, skills and items for each run, and then record them.

Obviously this only checks 1 target damage, but it's a place to start.

[marvalis]

Yes, I wanted to say "theoretical DPS" but thought well, I mean only whilst in combat, and people should assume this.

The question then becomes, looking at the game stdout or messages, how do you determine when you 'enter combat'?

* When you are within LOS of an enemy?
* Only when someone takes damage?
* You could take the tactical AI, if it takes a certain action, the player is in combat. For example: If the AI 'target' the player, then if the player used a skill then local enterCombat = game.turn then check a few turns later if anyone is taking damage else enterCombat= nil (you need a separate function for this but whatever you get the point).
* When you move from/away from an enemy (how to know when you flee?) (a good way to do this might be to check if the player killed it. If he does not kill it and goes out of combat (out of LOS, AI has no target, ...) then those turns should not be counted to calculate in-combat dps).

Point is, you need to write some kind of code to interpret this. Depending on how you define "out of combat" and "in combat" you can then calculate DPS, average dmg taken, and so forth. This is assuming we want to write some kind of program to calculate these values. Assuming we can use the info the game prints to do this.

'If' we can do this, then we can write something to send that data to the server.

How to code all that, sadly, I have absolutely no idea.

[gruevy]

I think that your ideas would be a good start, but I'm not sure that it would give all the information you'd really need to have to balance things. If you're going to go with online tracking of players for balance purposes, then you need more information.

For example, what's the single more important factor for archmages to beat the Shade? Is it damage? It is hitpoints? It is hp regen infusion power? If it was determined that 95% of archmages with x hps and y hp regen on infusions won, no matter what skills they had points in, then you'd have some useful information. Then you could look at other casting classes, like anorithil, and see if the same pattern emerges. If it doesn't, then what's different?

If you discovered that 80% of sun paladin who invested 20 points in a certain tree by level 30 made it to level 35, but that if they put those 20 points in a different tree, only 35% survived to level 35, then you'd have useful information.

But there's a big problem with tracking data of players for use in balance, and that's that right now, the gear that drops varies so widely between runs that it's very hard to say if it's a class problem or a crappy luck problem when a character dies. Until there are thousands of recorded runs of people trying every conceivable build, there won't be a way to even out the difference that gear makes in the numbers. Assuming we started tracking everything perfectly today, it may be months and months before there is really usable information for more than a handful of character builds, and the rest would be left out of the balancing discussion. That's why I propose testing in a controlled environment.

[Canderel]

there's a big problem with tracking data of players for use in balance, and that's that right now

The question then becomes, looking at the game stdout or messages, how do you determine when you 'enter combat'?

You do understand we are in a LUA environment,. I don't want the game to collect players' damage etc. (very big brothery IMO). I want to write a script that is "always" in combat, and you can select if it's 1v1 in a corridor, or 1 vs. many in an endless void etc.

So you kinda circumvent all the really impractical stuff to try to figure out how to program. Why do a lot of tough coding trying to "understand" what's going on in the game, if you can rather spend that time writing a balance testing environment.

Maybe simpler, is a Testing module, which will be played by players or a script (preferably script), but which has no randomness (you could possibly select the "monster level" which is accompanied with the results), a player/test can "cheat" towards any items/skills/levels they want, and the test bed stays the same. No XP gains.

Tests:

• A 2x1 corridor with you and a specific list of monsters. Mortal combat style, just more cramped. Continues until list depletes or char dies (to test 1v1 capability)
• Arena like: Huge empty space with monsters coming from all angles. 100 each wave, the monsters become increasingly difficult. Ends when list of waves are done, or char dies (to test crowd control)
• You can expand the tests if you need to gain different info (maybe one with a crowd coming from a big room to a corridor, and some walk the other way around, creating a 2v1 scenario).

[Canderel]

https://spreadsheets.google.com/ccc?key ... y=CJzN7_kH

I've created a spreadsheet, which at this time is intended for chars at level 20 - 25 (give or take, having had access to pretty much all skills the class have available in the game).

If people would like to help populate it, and add ideas/columns that add survivability to a class (or reduces it)...