DMB Final Episode - Bungie Interview with Mercules & Kyt_Kutcha
On the final episode of Destiny Massive Breakdowns, I (Legoleflash) had the pleasure of interviewing the previous hosts of the show, Mercules (Associate Weapons Designer) and Kyt_Kutcha (Contract Generalist Tester), to ask them questions from the community about their time at Bungie, Weapon Tuning, QA, and the Whetstone Exotic Mission.
For the interview, I'm also joined by Impetus, one of the hosts of the new flagship podcast for Destiny Massive Breakdowns: Podcast Versus Enemies. HUGE thank you to Bungie for allowing this to happen. It's a truly massive interview, and a fitting end to the Destiny Massive Breakdowns show. I hope you all enjoy it.
Destiny Massive Breakdowns Network:
Watch on youtube below, or catch up in your favorite podcast app.
Timestamps:
0:00 - Intro and Bungie Interview Guests
7:35 - Working at Bungie
14:20 - Destiny PvE Weapon Tuning
33:33 - Whetstone Exotic Mission
53:53 - Bungie QA
1:08:00 - Destiny PvP Weapon Tuning
Interview Breakdown
(PART 1 - Weapon Tuning)
NOTE: some questions & answers are summarized for reading.
Specific Questions to Kyt_Kutcha
Biggest change going from Podcast Host to working at Bungie?
Moving back from the community
Simply know too much. Safer to just say nothing!
What's your favorite daily thing you do while working on the team? What's something you are particularly proud of that you did/accomplished while on the team? Where can I find your work in game?
Favorite daily thing is loading the latest version of Destiny to see what's new
Sometimes twice a day loading a new version!
Spent a lot of time working on Deep Dives and Exotic Mission
Proudest achievement was pushing difficulty and add density in Deep Dives
That good players would be challenged as they went further
PvE Specific Weapon Tuning (Primarily Mercules)
Is the PvE ‘special meta’ and emerging prevalence of Specials intended and if not, will it be tuned in the future?
It is not intended. In S22 they will be correcting a bug that was introduced when Primary weapons went to unlimited ammo that causes more heavy to drop when running double specials.
Not directly nerfing double special but are keeping a tight eye on it. Double special is somewhat interesting when run with specials like Forerunner or Trace Rifles that have a custom ammo economy that blends the line between primary and special.
The idea of players being able to run, for example, a wave frame and a fusion and never run out of ammo is not really what they want to be the preferred way to play endgame PvE, so they will monitor how the heavy ammo changes affect the playstyle in S22. They imagine it will not have much of an effect so they will keep an eye on possible additional future changes.
Thoughts on the state of snipers in PvE, what is the philosophy behind them? For example, are they intended to be used against champs and majors or more designed towards bosses? Are they under performing in your eyes?
They feel like they are underperforming a bit, but there is a reason they are slow to move snipers upward. Because they have functionally unlimited range, if snipers become the best, they immediately invalidate a large number of other weapons and become one of the only things you need to run. Probably going to take another pass at them post Season 22.
Their intended purpose is primarily burst damage, so good for champs and shorter DPS phases while also having the ability to deal decent sustained damage. Need to be careful that snipers don’t get buffed enough that they begin to outclass heavy weapons. Similar to glaives and why they were tuned down, do not want special ammo weapons outclassing heavy weapons in terms of raw available damage.
Also has an effect on activity design, needing to make sure that players hiding far away and picking off enemies in safety is not a valid tactic unless it is specifically planned for. Do not want it to trivialize encounters.
What's the impetus behind making inventory size influenced by mag size, why does the influence pattern vary by weapon type, and do you have any hints about how it works (MGs and LFRs seem to function on how much rounding there is on the magazine size)? Also, do you have any plans to surface reserves in-game, given how consequential it is?
Asking about the deep, old magic here.
The general idea is that inventory size was originally fixed, but with magazine sizes being so variable you would end up with huge magazine sizes that left you with smaller fractions of a magazine in inventory. This felt weird so they changed it so that it was scaled based on magazine size so that could not happen.
As far as surfacing goes, they are investigating to see if they could do it because they understand the importance, but because it is a calculation based on calculations it requires far more work than surfacing the other stats did.
Are you going to buff any other legendary primary weapons in PvE? We already know about HCs, but what about autos, pulses, sidearms, etc.?
Note the HC buff in S22 is pretty substantial, in S23 they are testing a 10-15% buff to pulse rifles for PvE damage which if it tests well, they’d like to move into the mid-season 22 pass.
They think Sidearms are actually in a good state, they are just hurt by the dangers of close-range combat in high end PvE content, but as far as damage goes, they deal a lot. Scout Rifles not getting buffed any more anytime soon, have a similar issue to Snipers where if they are optimal players will sit in the back and plink away at things, and they also have unlimited ammo. Want to wait and see how the HC, Pulse Rifle, and double special heavy ammo drop changes affect the PvE primary ammo sandbox before looking at Auto Rifles.
Added a related note for players who are asking for their hand cannons to one tap red bars in GMs at their base level (i.e., no perks or damage buffs): “That is never going to happen. We are not going to let hand cannons one tap in a Grandmaster.” The Weapons team balances around the base sandbox, for example Hero Nightfalls and Legendary Campaigns. Anything below that your weapons will feel overpowered, anything above that they will start to feel underpowered.
They do this to avoid what they call an “arms race” with Activity Design. Activity Design has an idea for the preferred difficulty of the activity, and they start from the base sandbox and scale deltas and combatants up until they get there. If weapons were balanced for GMs, and HCs could one-tap red bars in them, then that content becomes much easier. So, then Activity Design comes along, and says this is too easy now, and they scale it up again. And that becomes the “arms race” where weapons scale up to meet endgame activities, and endgame activities scale up to outstrip weapons, and the cycle continues.
End up in a situation where people ask to go back to Shadowkeep precision damage, but a lot of primary weapons are not only dealing as much precision damage as they did in Shadowkeep, but they are also dealing substantially more body shot damage (approx. double). The sandbox has grown and then weapons grew too.
Want to stop that continuous cycle, and so now they balance for the base sandbox. They do look at GMs to make sure weapons don’t feel out of band compared to other weapons in GMs, but the focus of their balance is the base sandbox which is the upper middle tier difficulty activities.
Points out that saying you don’t think your weapons feel good in certain activities and content is a valid criticism, but not one that should be solved by scaling the weapons themselves up in power.
PvP Weapon Tuning (Mercules)
How do you handle tuning weapons that have a standard and an Adept version, when the effectiveness is drastically different between the two?
The goal for Adepts is to have them be maybe 5-10% more effective than the base weapon, not have them be significantly better than the base. A 5/5 roll of the base weapon should be close to a 4/5 of the Adept. Think they are hitting that goal pretty well. Most of the differences that do exist come from the Adept mods on specific weapons.
The actual biggest difference in effectiveness between the two versions is that the average skill of players using the Adept weapons is generally much higher than the average skill of players using the standard version, even though the average skill of the players using the standard version may be well above average when compared to the population as a whole.
The chart they showed in TWID the other week was the chart for high skill lobbies. People who were using the standard Immortal may be high skill when compared to the rest of the player base, but when they are isolated into high skill lobbies they are on the lower edge of that specific bracket, hence the underperformance shown in that specific chart.
If they had shown the chart of all skills, the standard Immortal is right on the line for 0% KoE, while the Adept Immortal is further elevated. They then combine those two values into a single value for “Immortal” and graph that out to see where it lies.
So, they do not look at the Adept and Standard versions as two different weapons for tuning purposes, but it is interesting to be able to see the split within a single weapon and see the performance differences based on average skill.
Is there any chance they will tune former lightweight hand cannons, either back to lightweights or adjust stats to reflect adaptive archetypes? They seem to really underperform (Waking Vigil as example).
They actually did already tune a number of Lightweight HC stats back in February of 2022, but the bigger issue is they have to consider the weapon source when they decide on how much to improve their stats. Waking Vigil is a destination weapon, so it is not going to have as good of stats as a raid or a dungeon weapon coming back.
Not every weapon from every source can compete with high end weapons, so some of the 150s that have come back as 140s fell victim to that. Generally, the easier the weapon is to acquire the less “cracked” the stat packages are going to be.
Also, if the stats on a weapon aren’t great that sometimes opens up the possibility of them putting really unique or strong perks on the weapons because the low stats will keep them in line.
Speaking of lightweights, it’s not a hand cannon but I know you love your redback sidearm, are there any new lightweight sidearms we have to look forward to?
There is a new Lightweight sidearm coming in S22 that is in the same vein as Redback but with some different stuff in the perk options, has been Merc’s go-to in PvP playtests.
Do you feel like optimal TTKs are where you want them to be in the current sandbox, or would adjusting them one way or another benefit the current Crucible sandbox? I think the main thing I’m curious about right now is “What is the design intent/fulfillment for TTKs to be as low as they are?” I don’t want them to be Y1 high but, I feel like us being able to take some weapons down to half a second and some even faster than that is a bit much.
There is a bit of hyperbole in that question, there aren’t a ton of weapons that can kill in half a second and if they can it’s on a conditional damage boost (Ravenous Beast, Kill Clip 120s, Adaptive pulses with Radiance). This is an intentional choice; they have tried to move away from things killing that fast because they think TtK values under 0.50s do not leave enough time for players to react or understand what is happening.
They think TtKs err maybe a little on the fast side, wouldn’t be opposed to shifting some of the stuff creeping up on 0.60s back a bit. At the same time, they need to have a pretty fast TtK for three reasons:
Moderately high special weapon uptime leads to a lot of instant kills, so primary weapons need to be able to kill pretty fast to deal with that. If they cannot, they would have to reduce the uptime of special weapons to compensate.
The presence of a lot of abilities that increase damage resistance or HP, and abilities that heal you. Need primary weapons to be able to output a lot of damage so those abilities do not get out of hand.
Destiny’s movement speeds are pretty fast, so if primary weapons don’t kill fast enough people can escape from engagements pretty easily and the combat loop becomes “run away if you get shot first” and that is not very fun
All that said, they have something coming in S22 for PvP called “Checkmate”. A new take on PvP with some modifications.
Not like D2Y1, but they have shifted TtKs a bit and modified ability cooldowns by a significant bit.
TtKs in normal PvP are between 0.60s and 1.00s, in Checkmate they are between 0.80s and 1.20s, and Hand Cannon TtKs did not change.
What is their baseline to determine, "Yes, this weapon is perfectly balanced and every adjustment to others should trend towards this"? I know that for many dedicated PvP players that baseline should be 140HCs.
They also consider it to be 140 HCs, in the current meta at the current point in time, specifically Rose and Ace of Spades.
They chart a slope on the Kills Over Expected chart, and basically if anything has higher usage AND higher effectiveness than those weapons it is a good indication it may need to be tuned down. Graviton Lance, NTTE, and Immortal are used as examples here.
Cautions that they look at high skill lobbies when utilizing this technique, because otherwise they would be nerfing things like Centrifuse which are highly effective in the center of the skill bell curve without being too effective at the high end.
Now that Comp has rank based MM (clarifies as long as you are within one major rank up or down from your skill, it purely uses your rank), they are able to do a good bit more top-down balancing where they can accurately see what is problematic in Ascendant lobbies, for example, and balance accordingly.
They look at the main Kills Over Expected chart showing all skill levels first to find outliers, then go to look at the high skill KoE chart. If something is an outlier in both places, then it is an indication that it may need to be tuned down.
Is there a common resilience level you balance weapons for? Many weapons seem to use resilience checks to keep them in their place. Is there a “go to” resilience number for that?
They consider base resilience to be 60, which is 192 HP (70 health and 122 shields).
Forgiveness checks can be anywhere on the resiliency chart, just a bonus that you are harder to kill.
If a TtK drops dramatically (Adaptive pulses rifles used as an example) it needs to be at a very low resilience level so that people can build above it if they want to without much investment.
If TtK increases and pushes a weapon out of meta, that should be at higher resilience levels, because otherwise everyone hits it, and the gun is just worse at base.
When they change damage on a weapon, they run damage checks at 10%, 15%, 20%, and 25% damage bonuses to see how it changes TtKs and forgiveness in PvP. The activators for 25%+ are generally pretty stringent and so they are okay with it shifting weapon TtK by a good bit. For 10% bonuses that are on high uptime activators, they do not want it to do much to a weapon’s TtK unless the weapon isn’t very strong at base (again used Adaptive pulses as an example). 15-20% they are more case by case on how they look at them. They won’t put Golden Tricorn on something that is already strong where it is going to lead to a huge decrease in TtK because the first activator is too easy.
Generally, they are okay with players getting strong effects from stacking multiple damage perks at the same time, except in rare cases. Mentioned not being able to buff Gutshot Straight’s damage bonus because combining it with Radiant could allow 140 HCs to kill in 3-body shots.
What metrics do you use to determine what is the strongest in PvP?
Use a bunch of metrics, there is no single metric to say something is the strongest. If they only used KoE, then Sweet Business would be considered the strongest weapon in PvP at the moment.
Community feedback, playtest data and feedback, personal experiences playing the live game
Usage - It is a tool that they use but it is far from the only tool. It is one of the lower tools comparatively, but it has to be acknowledged because it is the metric that is most visible to players, and a good indicator of how often players experience the thing in game.
Net Fun - How much fun the person using the weapon is having versus how little fun the person dying to the thing is having. Net Fun is not linear, it ratchets down exponentially as usage increases. Cloudstrike used as an example where it wasn’t so bad before when it was rare to experience, but now that it is everywhere the Net Fun has shifted to be negative overall.
When usage is low, weapon effectiveness is highly variable because highly skilled players using the weapon in niche builds will boost the KoE, but as more players start using the gun and the average skill of the user comes down it should bring the performance of the gun down so that it normalizes. If it does not come down, then it is an indicator that the weapon is too strong.
Kills Over Expected is very useful because it graphs things out in a nice easy to see slope. Makes it easy to chart a line and say, “This thing sees more usage than this other thing and yet the effectiveness is way higher. Either this thing needs a buff, or this thing needs a nerf.”
Do you feel that Hand Cannons have been power crept in the Crucible?
They keep track of all the balance changes they make in a historical document so they can see where guns were a year ago, for example, compared to where they are today.
In terms of actual weapons that have power crept HCs, it’s the Immortal and Graviton Lance. Those are the two things that have surpassed HCs.
The other stuff that has been changed in the last year:
High Impact Scouts got their damage buffed by 2
Auto Rifles have been buffed a number of times, but outside of the center of the bell curve they do not appear to be treading on HCs.
Pulse Rifles
Nerfed four times - Low handling, range x2, nerfed aim assist
Also nerfed NTTE x3
Buffed - Lightweight base damage by 1, Adaptive precision damage by 1, high handling
Don’t think that either Lightweights or Adaptives have power crept HCs in the meta
Hand Cannons have not been nerfed in quite some time, and they buffed 180s
SMGs have been nerfed a lot. Range twice, Lightweight base damage, Ikelos directly, then Aggressive damage, then Immortal directly, then Target Lock.
The only buffs they have gotten were Adaptive precision damage and Precision base damage, but that was paired with a nerf to Shayura’s as well.
Sidearms did get the AA falloff buff, but they aren’t currently meta, and Forerunner didn’t get that buff.
Don’t think HCs have been power crept. Think what has happened is the state of PvP has shifted as highly skilled players match each other more frequently. People are playing their lives much harder than they have in the past and are taking games more seriously, and pulse rifles mesh well with that playstyle. Pulse usage has come up even though technically most pulses are weaker now than they were a year ago.
If you look at the top 25 most used primary weapons in high skill PvP lobbies from the last two weeks, they make up about 70% of all primary weapon kills in those lobbies. The top 50 weapons are 80% of kills, the top 100 are 97%.
Stats from the top 25 weapons:
The most effective primary weapon in high skill lobbies over the last two weeks was Spare Rations
5 of the top 10 most effective primary weapons were HCs - Spare Rations, Ace of Spades, Rose, Round Robin, Hawkmoon
Adept Immortal is the 7th most effective primary weapon in high skill lobbies, although it is the most used. Historically, Immortal is not even in the top 10 for most unbalanced primary weapons. Hard Light once got 30% of all weapon kills by itself. There was a period of time where HCs alone had more kills than every other primary weapon combined in PvP. If you added just Felwinters to the HC numbers, they had more kills than ALL other weapons combined in PvP. DMT had less than half the usage of Immortal and was getting the same number of kills.
Immortal’s biggest sin currently is usage. Almost all SMG usage is consolidated in Immortal, and so it feels oppressive and overused. There are only 3 SMGs in the top 25 primary weapons in high skill lobbies, including Immortal and Immortal (Adept).
Comparatively, there are 9 HCs. HCs only see slightly less usage than pulse rifles and cumulatively they are also the most effective class of primary weapon at high skill. But because their usage is spread out across a larger number of weapons, it doesn’t embed itself in your brain the same way Immortal does when you hear its firing sound over and over.
In the top 50 primary weapons there are:
17 HCs
12 PRs
7 SMGs (Merc said 3 in the episode, clarified the correct number after)
5 Auto Rifles
6 Scout Rifles
3 Sidearm
1 Bow
They have said that the best primary meta is HCs at the top, Pulses right underneath, then ARs and SMGs, then scouts below that as they don’t really want an AR or Scout meta. And that is kind of where things are at, it just takes a little bit for perception to catch up to the reality.
This is not to say they think Immortal is fine where it is, they don’t think that. But the Season 22 changes directly affect Immortal by nerfing its zoom with Rangefinder. The changes also affect Shayura’s, which is data wise and in Merc’s opinion is actually the most effective SMG, albeit the availability is much lower.
Can we get a meta shift to underperforming weapons? I'd love to see 180 hand cannon/360 auto meta and glaives be the close range special meta.
All of these weapons are getting buffed in either S22 or S23. 180 HCs are benefitting a lot from the range/zoom changes, and 360 ARs that aren’t at 19-20 zoom are as well.
Do not want a 360 AR meta though, because in order for ARs to compete at the highest level of gameplay they have to kill much faster than other options that can peek shoot (like they did when 600 RPM ARs were meta).
180 HCs are hard to tune, because you can really only kill in 0.67s or 1.00s. Try not to tune RoF on weapons because very small changes have a very big difference. For example, the difference between 600 and 450 RPM ARs is only a single frame at 30 FPS, and the difference between a 600 and a 720 RPM is only 1 frame every other shot. So tuning RoF can make a gun feel very good or very bad, so they usually do it on perks that are temporary.
Do not want glaives to be close range meta. They are getting a big overhaul in S23, increasing their projectile speed significantly. Looking at letting shield charge recharge slowly when the glaive is held, looking at reducing the delay after firing before you can melee, and looked at tuning the shield DR down in PvP to prevent it from becoming oppressive if the uptime comes up.
Want glaives to feel better to use but not feel bad to play against.
How do you feel about Fusion Rifles' place in the Crucible at the moment? Will the zoom changes put fusion rifles more in line with how zoom works on other weapons, or will it only change how damage falloff distance relates to zoom?
At the high end, they think fusions are too strong in PvP, in large part due to a couple of perks they are going to be addressing.
Fusions are one of the rare archetypes where they are almost perfectly balanced across the sub-families, there is near equivalent usage and effectiveness among Rapid Fires, Adaptives, Precisions, and High Impacts.
Their usage has come down a good bit from where they were in the past, which is good, but they’re going to nerf Under Pressure in S22 and reduce the maximum accuracy bonus it can grant. It is probably the best perk on fusions and is pretty out of band for a perk that has near 100% uptime in PvP. Also, in either S23 or S22 Mid-Season they are going to bring Kickstart down a bit.
Fusion Rifles in the live game have a custom modifier that substantially reduces the benefits provided by zoom, which includes damage falloff distance, aim assist falloff distance, stability, and accuracy. In S22, they removed this custom tuning and set the damage falloff to be slightly lower than where it is in live. So, it is a small damage falloff nerf and a buff to the other effects, and that is what led to the Under Pressure change. What this means is that in S22 a fusion with Under Pressure will feel slightly worse than it does right now, but any fusion without Under Pressure will feel better than they currently do. They squeezed the top edge down and lifted up the lower fusions that didn’t see play because they couldn’t roll Under Pressure.
Will Cloudstrike be looked at or is it in an acceptable spot?
In Season 22 they are going to reduce the radius around the player who gets headshot that is lethal. Going from around 5m to closer to 3m. Pretty substantial reduction to the splash damage.
With balancing Legendary weapons, do you individually tune weapons with stats outside of the listed stats we see in-game? For example, if two weapons in the same archetype have the exact same range, stability, aim assist, handling, and recoil direction, are there other stats in the background making each legendary weapon feel different?
There are sometimes, but it is not something they do intentionally anymore, don’t really want there to be an invisible stat in the background that determines if a weapon is better or not. Want you to be able to do head-to-head comparisons and have those be accurate. Ikelos SMG was an example of one that had it and it was removed.
One thing that is custom tuned are the sights, optics, scopes of weapons. If those are unique, they are made by hand, so things like depth of field, blur, sight picture, size of the sight, the sight picture, etc. So, there is a difference in feel there, but it shouldn’t directly affect performance. Examples of some players not liking Bygones sight while other players love it, or SUROS HCs.
Another thing is that weapons have different recoil patterns. Different sub-families can have different recoil patterns (for example Lightweight HCs recoil differently than Aggressive HCs do, but all Aggressives should recoil the same).
Some exotics have lots of custom tuning though.
What are the plans for bows coming in season 22?
There was an issue where the projectiles for bows did not have fast enough velocity to be registered as hitscan at longer ranges, which was leading to weird hit registration, especially on Lightweights
Increased the projectile velocity of Precision bows so they should remain hitscan at longer distances, and then matched the projectile velocity of Lightweights up to it, so that they should see a massive improvement as well.
Community members noticed that bow reload perks stopped being effective above a certain level, they investigated and found that there was a lock on the animation preventing it from running faster than a certain limit. Reduced that limit so perks will continue to have an effect.
Interview Breakdown (part 2)
(NOTE: some questions & answers are summarized for reading)
Whetstone Exotic Mission (Kyt_Kutcha)
What's it like for that Exotic Mission to finally be revealed?
Relieved that the secret is out! The biggest thing though is that we share in the excitement of the players as they discover and explore this new thing that so many hours went into making. It takes a lot of work to create a hidden mission like this, and even more work to drop enough clues to it without giving everything away. Following along with players as they discover and explore it is a ton of fun!
Will we see more activities like Deep Dives and hidden missions inside them (like Wicked Implement) in the future?
Have to wait and see, BUT every exotic mission is very important to the team and for this one, a sense of discovery was really important.
IF they do something like this again, there will definitely be some learnings taken from this effort.
How do they balance the kind of enemies that appear vs the timer? Also do certain enemies lend themselves better to a timed mission than others?
The type of enemy seemed totally dependent on the story and lore of the encounter, then the team went from there. * From a testing perspective, making sure certain types of adds didn't take up too much time needed for bosses was mentioned.
DPS is mechanically earned, and they didn't want too many distractions.
Thinking about Taken Goblins specifically.
How was the balance of the encounter time and boss HP determined, to achieve a certain level of difficulty?
A lot of playtesting and tuning happened to determine a sweet spot for completing the encounter. Rough estimates were that we wanted the Knight boss to be defeated with around 8-7 minutes remaining and then the Tormentor to be defeated with under 90 seconds remaining.
What was the thought process behind putting the Exotic Mission inside a matchmade activity? What were the concerns?
The Whetstone encounter was originally thought up to be a spiritual successor of the Black Spindle mission in Destiny 1.
We were concerned about players not enjoying being kicked to orbit or having to redo Deep Dives to access it, but on top of having to reach the door in Lost to Light within the time limit, players had to wait in real time for the mission to be available, so this felt pretty comparable to that experience.
There was a LOT of discussion about this on test, in terms of how to make sure it was fun and worthwhile in those conditions, not to mention just that it worked at all. We spent a lot of time looking not just at difficulty, but also at various griefing scenarios and trying to remove as much downside as possible from the matchmaking in Deep Dives as a whole.
Ultimately though, Destiny is a game you play with other people and matchmaking in that regard was a huge upside for the activity. This was also an experience that all the teams involved learned a lot from, and those learnings will be carried forward in future experiences.
Confirmation on catalyst drop requirements. Can it drop from a normal deep dive or does it have to be a tier 7?
Tier 7 Deep Dives is a guarantee, however it is still fairly likely on slightly lower reward tiers.
Was it hard to get approved? Are you worried players will come to expect these types of secret missions?
A sense of mystery and the opportunity for players to uncover secrets is something that’s central to the experience of Destiny. So no!
The colors in the labyrinth... What do they mean?
While testing inside of the maze encounter players would consistently get lost and lose their bearings, which caused them to waste extra time. They ended up coloring each quadrant of the maze a different color so players could differentiate, as well as raising and lowering sections so they are more distinct.
---
QA at Bungie Questions (Kyt_Kutcha)
How many iterations of an idea do you typically work through with the dev team that designed the thing you're testing?
Anywhere from 1 to dozens; there's no typical number. Sometimes they nail it on the first go, and other things just don’t work out the way they are, especially early on, and they’ll end up morphing considerably, or being replaced, or being cut.
Curious as to what the blank process looks like before something is identified and tested. Are you trying to recreate a bugs list. Or just playing in unconventional ways. Any other insight?
Yes, both of those! Like any agile software QA process, there are a lot of standard user stories and test cases that get reused every single time. We know all the common stuff that can go wrong and we always cover the bases. We also have some great tools for testing, which are getting better and better over time, so we’re always getting better at spotting things like OOE issues, missing physics, and other common game design problems.
Playing unconventionally is what we in the testing world call ad hoc or exploratory testing, and the version of that most people would think of (just go play and try weird stuff) is generally a pretty small part of testing. That's because rather than doing things randomly, we do a lot of directed brainstorming for activity specific concerns and generate user stories around those concerns.
So some basic examples include stuff like:
Can we use Strand to go too fast or bypass a trigger and break the activity? Does Grapple make carry object objectives, of which there are a few in Deep Dives, trivial, or is it a fair trade off?
What happens if we kill all the enemies extremely fast? Wiping the floor too fast can sometimes cause something weird to happen with objectives that trigger from clearing a wave.
How much is "fun" a factor in QA? Is it all just bugs? Or is how satisfying a feature is to use a part of the feedback loop and testing?
Fun factor is a huge part of the QA process, for sure. It’s one of those things that’s always in the back of our heads, even if we’re specifically looking for bugs or executing other tests.
There are also a lot of open playtests internally that are specifically centered around getting feedback from folks in the company that haven’t played the feature before, or at least not as much as the test team assigned to it.
(Sounds like they have a lot of fun themselves with this! I'm super jealous.)
What's the biggest struggle? What's the version control like?
Version control is very rigorous, as it has to be with a game as complex as this. In test they get daily change lists to keep us informed about what’s new or recently fixed, which is often a guide for what to check on that day.
Always double checking to see if a pesky old bug has returned or if a fix for one bug has any unintended consequences – regression testing is a big deal, as with all software development.
Highly engaged during repetitive tasks is one of the hardest parts, and sounds like a key skill for the team.
What do the test scripts / procedures look like? Are there QA people that specialize in breaking stuff? How much time is spent on balance testing vs feature/bugfix testing?
They would look extremely familiar to anyone doing any kind of software QA.
Lots of spreadsheets (perfect for previous DMB hosts)
Hard to point at things they contribute to because the players never see the results. They fix these things before the players ever see them.
PVE balance is often something that is dialed in along the way based on the activity’s goals, and then gets a stronger focus once the activity reaches a stable state. Design also has a lot of data and feedback from past activities that they can use to level set when making a new activity, so that is a big help. Ultimately, balance is also a subjective thing, and they know that it’s never going to be perfect for everyone. The aim is to hit as close to the target for each activity as possible, and then to learn and improve with every release.
I highly recommend listening to the full show if you're looking for more personality and insight!