Hellsweeper VR: Implementing Dynamic Foveated Rendering for PSVR2 and Cross-Platform Optimization

On March 28, 2024, our team crossed one of the most challenging milestones in the development of Hellsweeper VR. Six months after its launch, we managed to implement Eye-tracked Dynamic Foveated Rendering (ETDFR) for PSVR2 as part of the Wrath Update. The resulting performance boost then enabled us to drastically improve the visuals.

In the process, we also managed to enhance the graphics on other platforms (i.e. PCVR, Meta Quest, Pico) because redoing the entire pipeline to enable ETDFR for PSVR2 meant we could start fresh and revisit the other builds, too.

This article covers what we learned after half a year of breaking our game apart in order to rebuild a better version of it.

We are not Experts, just Learners.

It’s important that we preface this guide by saying that every project is unique, and there are often multiple ways to tackle the same problems.

What we are sharing is based on our own experience, which may not necessarily be the most optimal workflow. So, we’d love to hear from other developers who might have better or different approaches!

With a small team coming off an older Unity workflow, we had to figure out a lot of things by trial and error, going through platform and engine documentations and studying tutorials created by other amazing devs online. We’re grateful to have found our way out of the woods, but we probably wouldn’t want to go through it again.

By compiling this guide, we hope to help other indie VR devs caught in the Unity transition or facing optimization hurdles.

---

Our Main Challenges

Hellsweeper, at its core, is a fast-paced action game similar to our first VR game, Sairento. Players fight their way through hordes of demons while performing high-octane actions like somersaulting, wallrunning, double-jumping and more, and mastering various types of weapons and magic.

There's a lot going on in there.

In short, there’s a lot going on, especially coupled with the realistic art style that we were going for.

But unlike Sairento, there are elements that made it more challenging to optimize:

• More organic-looking surfaces due to the “hellish” theme. Flat surfaces are usually easier to optimize.

• Loads of magic in the mechanics. Particle effects (VFX) increase performance requirements.

• More enemy types and locations. This means more textures and materials.

Particle effects are pretty but they can easily wreck the frame rate or even cause crashes.

Below are some key information about our build before we implemented ETDFR:

• Unity Version: 2021.2.27

• Render Pipeline: Built-in / Standard Pipeline (SRP)

• Render Passes: Multi-pass Rendering
  

Based on the available options back then, PSVR2’s eye-tracked dynamic foveated rendering (ETDFR) plugin required at least Unity 2022.2 and the Universal Render Pipeline (URP).

We recommend the Universal Render Pipeline (URP) for most VR developers because URP will be our first render pipeline to support some of PS VR2’s unique features, such as foveated rendering and gaze tracking. (Unity Blog)

This was VERY BAD NEWS because we were using the Built-in / Standard Pipeline, and switching pipelines is a massive undertaking.

It's important to note that we started coding Hellsweeper at the start of 2020. During that time, URP was quite unstable and so while it showed a lot of promise, it was just too risky. The specs and requirements for PSVR2 were also not available yet. We didn’t know what we didn’t know. So, when it finally happened, we were one of the studios caught during the transition.

Updating Unity and Implementing URP

Short of rebuilding the entire game, updating to a newer Unity version and switching from SRP to URP was the only option to move forward.

First, we switched to Unity 2022.3.7 because our version of Unity wasn’t compatible with PSVR2’s foveated rendering plugin. The switch messed up some of the physics in the game. 

Then, we converted our render pipeline from SRP to URP. The transition broke a LOT of things.

The next six months was basically this.

❌ Shaders

Many of our original shaders became incompatible with URP, resulting in a sea of purple materials. Purple enemies, purple VFX, purple waters… We despised purple for quite a while. We had to either update the shaders or search for compatible ones that could deliver similar or even better results. We used mostly third-party shaders as we didn’t have an in-house shader expert who could code them from scratch.

Broken materials everywhere! We despised purple for quite a while… 

❌ Lighting Calculations

Upgrading Unity’s version often brings significant alterations to light baking calculations. As a result, we had to revisit and redo all lighting for all platforms. 

Colors became dull and lighting flat after converting from SRP to URP.

❌ Plugin Compatibility

The shift to URP rendered many third-party graphical plugins incompatible. Like the shaders, we had to search for suitable alternatives and tweak them according to our project.

❌ New Project Settings

This isn’t technically ‘broken’, but it needed some getting used to. URP introduced project settings that differed substantially from SRP. We had to learn our way through new processes.

Fixing the other ugly bits...

The whole transition from SRP to URP was a very labor-intensive process. But on top of unlocking compatibility, another good thing came out of it: as we needed to search for errors, we had to review every single corner in the game. In the process of fixing errors, we were also able to improve some textures and 3D models — either to make them prettier or to further optimize them.

Replacing horrendous, over-optimized models with new ones that had actual shapes.

Increasing the resolution of the most commonly-used textures (e.g. walls).

The Final Stretch: Implementing Eye-Tracked Dynamic Foveated Rendering

Once most of the visual anomalies caused by the URP transition were resolved, implementing either FFR or ETDFR was actually pretty straight-forward.

We tested FFR first and while it improved performance by about 10–15%, it wasn’t enough to make a meaningful difference.

We initially had issues getting ETDFR to work (due to an embarrassing noob mistake that we’ll talk about later), but we eventually made it happen. The main challenge was mainly in finding the right settings that could give us enough performance boost to use for ‘beautification’ while maintaining just enough to keep the game stable.

We created a Dev console in our development build that allowed us to test different settings on the spot.

Final Results: A 15-35% boost in performance!

Here are some of the key visual upgrades that we were able to implement after the performance boost from ETDFR.

Here’s an overview of the enhancements resulting from all the combined optimization techniques we implemented in addition to ETDFR:

• Render Resolution: We managed to boost the core render resolution from .75 (original) to 0.9 (week 1 patch) to 1.5 (after ETDFR). This was the most noticeable improvement!

• Anti-Aliasing: To address the jittery lines around 3D models in the game, we turned on Multisample Anti-Aliasing (MSAA), significantly reducing visual artifacts.

• Shadow Quality: We improved shadow quality by increasing the resolution and incorporating the highest level of shadow cascades. We would have loved to implement soft shadows, but they’re too resource-intensive.

• Increased Particles: We managed to bring our PCVR (VFX) to PSVR2 with higher particle counts and more intensive effects.
  

Lastly, one of our biggest challenges with cross-platform development was finding the right shaders. Some shaders that worked well on both mobile and PC didn’t work on PSVR2. We had to spend a lot of time testing and experimenting with different shaders to ensure they performed well and looked good on all supported platforms.

Here's a comparison of the boost we got from FFR vs ETDFR. Worth it!

Fixed Foveated Rendering (FFR)	Eye-Tracked Dynamic Foveated Rendering (ETDFR)	Additional Optimization Requirements
+15% performance gains	+35% performance gains	-----
MSAA x 2	MSAA x 2	Accumulated effort
	Increased resolution from x.75 to x.9 then finally to x1.5	Accumulated effort
	Improved real-time shadows	Decreased shadow casters
	Increased particle count for VFX	Reviewed and optimized each VFX further

Improving the Game's Stability and Performance

Optimization is an on-going process. Since the release of the Wrath Update with ETDFR, we've rolled out three hotfixes addressing crash reports on PSVR2 among other things. We traced the issues back to incompatible shaders, specifically the ones used for hands-only avatar and the hair/fur.

The feedback has been positive, and we'll continue to monitor reports related to the game's stability as we work on new content.

Dynamic Foveated Rendering is awesome, but it is not a miracle cure.

While FFR or ETDFR are wonderful tools that can significantly free up performance requirements, in most cases, they should not be used as the main method to optimize games that are hitting the performance limits.

We went through multiple rounds of optimization, using several other methods that we’ll cover below, during the six months of transition. We were constantly working on optimization — eating, breathing, and dreaming about it. It wasn’t sexy.

---

Squeezing Every Bit of Performance Boost.

Some of these optimization methods apply to both mobile and high-end graphics (PC and PSVR2), while some apply only to mobile. We decided to compile all of them regardless of platform since if you’re developing for various devices with different computational capabilities, it’s important to be familiar with all available options.

We were constantly removing and adding things, to find the best balance between performance and visuals.

Audio Optimization

A lot of devs focus on cutting the load through visuals, but we realized that a very important yet not-so-commonly discussed component is audio optimization. This greatly decreases the amount of memory used, which is always the main cause of crashes. 

The balance, of course, is trying to hit the performance rewards while keeping the audio crisp. This in-depth guide about Audio compression in Unity helped us a lot.

Using Packed Maps

We also used 2 packed maps per material for enemies so that we could decrease the amount of textures used. 

• Roughness, metallic, and AO (Normal Material everywhere in the scene)

• Burn, Freeze, Electrified, Bloody status (Additional for Enemies)
  

Unity has a limit of 7 texture calls per material, and going beyond this limit can cause issues when compiling. Without using pack maps, our enemies could have ended up needing at least 7–9 textures!

Look at this mess. Eight materials for one single character!

We also browsed through all our models and selected which ones were necessary to use heavier shaders and which ones could make do with mobile ones.

Use Occlusion Culling Wisely

Before the URP transition, we decided to break our maps apart to implement occlusion culling. Essentially, occlusion culling allows us to not render objects that are not visible to the camera because they are blocked by other objects. However, this did not bring the improvements we expected; in fact, some maps ended up with even more lag.

Why? We overlooked a crucial detail: Since we have many arena maps, players can see most, if not all, parts of the map from any spot, making occlusion culling far less effective.

Occlusion Culling worked for maps with large and tall walls that covered other sections of the same map.

Having said that, this still improved performance for a few maps that had large walls like the Vault tunnels and Hangman’s Caverns.

Realtime Shadow Performance

When we launched the game back in Sep 2023, we had to drastically reduce the resolution of real-time shadows to hit PSVR2’s performance standards. This led to pixelated shadows that looked more at home in tetris.

Shadows are costly but getting them right is important for VR immersion. Along the way, we discovered that one of the bottlenecks stemmed from the high number of shadow casters. Reducing the number of shadow casters improved the effectiveness of occlusion culling, decreasing the polycount of the models responsible for casting shadows, essentially enhancing performance.

We managed to increase shadow resolution from 512px to 2k.

Although this guide is mostly for mobile, it gave us some ideas on how to get started. Sometimes, online tutorials didn't exactly provide the direct solutions but they helped us look at problems from different angles.

With the performance hurdles out of the way, it was time to make the shadows look good. At first, the only obvious solution was to increase the resolution to 4k and enable soft shadows. But this led to a substantial dip in performance.

Our breakthrough came when we stumbled upon a simple guide about using cascades for shadow optimization. We knew this method well — we’d used it on PC, but it had slipped our minds amidst all the PSVR2 chaos.

From 512px, we managed to increase our shadow resolution to 2k with max cascades without hurting performance.

From Multi-pass Rendering to Single-Pass Rendering

Initially, we used multi-pass rendering for PCVR and PSVR2 without any issues. However, after switching to URP, we encountered problems: many shaders stopped working, rendering only in one eye, and the framerate for PSVR2 dropped below 90fps.

So, we decided to switch to single-pass rendering across all platforms. This not only resolved the problems but also provided a much-needed performance boost.

Note: As of March 22, 2024, a newly released Unity SDK seems to address an issue that stemmed from Single-Pass Rendering not working accurately with ETDFR on PSVR2 (officially shared in the private PSVR2 dev forums), but this wasn’t relevant to us as we never faced this problem. We’re not familiar with this version of Unity so we can only assume that this might be related to more complex features rather than baseline compatibility.

Additional Resources

Wobbly Environment after ETDFR — Don’t Forget to Calibrate Your Headset. 

Yup, noob mistake.

When we first enabled ETDFR, the visuals became noticeably clearer but also started to wobble especially around the peripherals. We couldn’t figure out what triggered it and every fix we tried just crashed the game.

Maybe we messed something up with ETDFR, or maybe it’s just Unity being Unity?

When all else fails... blame Unity.

We were close to throwing in the towel and just settling for FFR. But after six months of hard work, giving up without one final push just didn’t feel right. So, we fired up No Man’s Sky to admire the view, and thought, ‘If they can do it, why can’t we?’

Back to the drawing board, we eventually stumbled upon reddit discussions about wobbles in RE8. And what do you know? It turned out we just hadn’t calibrated our headsets properly!

Lightmap Improvement (for Mobile Only)

Switching to URP messed up the baked lights for mobile. None of the various workarounds improved the lighting until we tried the Bakery plugin. This tool proved to be much better than Unity’s built-in solution. Even when reducing the lightmap resolution, Bakery manages to preserve details, including baking ambient occlusion effectively, unlike the standard SRP, where details often get lost in such scenarios.

Old but still Gold

Back when we were working on our first game for mobile VR, Sairento Untethered, this was a video that helped us a lot. Although this video is about five years old and may be outdated by today’s standards, it was still relevant when we were optimizing Hellsweeper for mobile VR.

---

In Summary

If we could rewind to four years ago when we first started working on Hellsweeper, we probably would have still started with the Standard Pipeline because the Universal Render Pipeline was unstable, but we would have definitely made the switch earlier.

Occasionally, we also still toy with the idea of ‘What if we had developed Hellsweeper using Unreal instead?’ But this would have meant letting go of the majority of our devs, which was never an option.

We’re grateful for the experience and the lessons learned, but the last six months have been mentally and physically challenging. We had to keep pushing forward despite knowing that our efforts might not make any difference in the end. However, as a small studio that owe our humble beginnings to the support of the VR community, delivering on what we promised was what kept us going.

We know that nothing can fully replace launching with the expected quality, but the support and warm messages from within and beyond the PSVR2 community after the Dynamic Foveated Rendering update have made it all worthwhile.

To those who pushed us to do better, those who cheered us on, and those who came back to give Hellsweeper VR another chance — thank you to all of you.

So, what's next for Hellsweeper?

Let's just say, we're not done cooking yet ;)