Epic Online Services Environment Selection

 When starting up a game using Epic Online Services, your game logic needs to select the correct environment to use.  Epic generally provides 3 environments: Development, Staging, and Live.  Each have their own Sandbox IDs and Deployment IDs.  Generally speaking, you should include configuration data for all 3 environments in your build, and based on launch arguments from the Epic Launcher, decide which environment to actually use.  In all cases, when running your game in a local development environment and not from the Epic Launcher, it will use the default configured environment.

For Unity, Epic recommends the PlayEveryware EOS plugin.  Epic has actually taken over development of the plugin.  You can follow the official instructions to enter your environment configuration values into the project.  As long as you enter all 3 environments, when the game is run, the plugin will check the Sandbox ID that was passed in via the Epic Launcher and match it to one of your existing environments, then use that for the game.

For Unreal Engine 5.1 and newer, Epic recommends using their EOS Online Services plugin.  However, the configuration here does not seem to allow multiple environments for the same platform.  You can only configure one environment per platform, and must make a different build for each environment.

However, the older EOS Online Subsystem in Unreal Engine 5 does allow for multiple environments to be configured.  It will then select the correct one based on the Sandbox ID from the Epic Launcher command line, as the Unity plugin does.  In your Engine config, set up the 3 environments:

[/Script/OnlineSubsystemEOS.EOSSettings]
DefaultArtifactName=<live artifact name>
+Artifacts=(
    ArtifactName="<dev artifact name>",
    ClientId="<client ID>",
    ClientSecret="<client secret>",
    ProductId="<product ID>",
    SandboxId="<dev sandbox ID>",
    DeploymentId="<dev deployment ID>",
    EncryptionKey="<encryption key>"
)
+Artifacts=(
    ArtifactName="<stage artifact name>",
    ClientId="<client ID>",
    ClientSecret="<client secret>",
    ProductId="<product ID>",
    SandboxId="<stage sandbox ID>",
    DeploymentId="<stage deployment ID>",
    EncryptionKey="<encryption key>"
)
+Artifacts=(
    ArtifactName="<live artifact name>",
    ClientId="<client ID>",
    ClientSecret="<client secret>",
    ProductId="<product ID>",
    SandboxId="<live sandbox ID>",
    DeploymentId="<live deployment ID>",
    EncryptionKey="<encryption key>"
)

Each environment should have identical values, except for the Artifact Name, Sandbox ID, and Deployment ID. 

Note that when running your game locally, you can add the "EOSArtifactNameOverride" command line argument to forcibly select the environment you want to use, regardless of the defaults set in the config file.  This will not work in the live game running from the Epic Launcher, though.  Example: <path to game>/<game>.exe -EOSArtifactNameOverride=<dev artifact name> -EpicPortal

(Also note that for Unreal projects, adding the "EpicPortal" command line argument signals Unreal that it can initialize the eCommerce system if you need to test that locally.)

In Unreal 4.27, the Online Subsystem logic does not select based on Sandbox ID.  It will try to select based on the Epic Launcher "EpicApp" argument.  However, that argument is the Artifact ID of the game.  That ID does not change between environments.  You would have to set up different Artifacts in the Epic Dev Portal for development, staging, and live in order to use this feature.  Without modifying code, you will again need to make different builds for each environment.  In this case, you can change the "DefaultArtifactName" config setting to match the particular environment you are targeting.

If you want to modify Unreal 4.27 source to detect the environment via the command line, check Engine/Plugins/Online/OnlineSubsystemEOS/Source/OnlineSubsystemEOS/Private/EOSSettings.cpp.  Compare it to the same file in the source for Unreal Engine 5 and copy over the logic for selecting the artifact settings based on the Sandbox ID.

As a side note, whenever your game is run the from the Epic Launcher, it has multiple command line arguments.  You can view these arguments in Windows via several different methods.  We recommend Process Explorer.  A typical command line will look like this:

<path to game>/<game>.exe -AUTH_LOGIN=unused -AUTH_PASSWORD=<password> -AUTH_TYPE=exchangecode -epicapp=<artifact ID> -epicenv=<Dev/Stage/Prod> -EpicPortal -epicusername="<username>" -epicuserid=<user id> -epiclocale=<locale> -epicsandboxid=<sandbox ID>

Jenkins: Create a Mac Node

Some of our Unity projects require building on a Mac for the macOS and iOS platforms.  We have most of our build processes automated through Jenkins to build on Windows, but we recently went through the process of setting up a Jenkins node on the Mac to handle the additional platforms.

There is an existing guide, Configuring a macOS Machine as a Jenkins Node by Jishnu Nambiar, that is a very good starting place.  However, we still ran into several issues that this guide does not address, likely because we are working with macOS Sequoia 15.3.2.

Mac User Setup

We will also use Jishnu Nambiar's guide, How to Connect to Your Mac Remotely Using Windows/Linux.  There is one issue we were unable to solve: we had to create a second Mac user as an administrator, because otherwise the SSH login would not succeed due to a low-level block by macOS.  There may be a way around this so we don't need to log in with an administrative user, and we will update this guide if we find it.

In addition to making sure the new user you create has administrator permissions, you also need to enable Remote Login in the same location you enable Remote Management.

Jenkins connects to the Mac via SSH, so permissions run through that system.  In this case, we need to give disk access to the SSH server on the Mac.  Go to System Preferences, then Privacy & Security, then Full Disk Access, and grant the permission to sshd-keygen-wrapper.  You may need to try connecting to the Mac via SSH once and try to make a modification on disk to trigger this.

In general, you can test login issues using an SSH client like PuTTY for Windows.  Point at the IP address of your Mac and start a session, then log in with the new user name and password.  You can also set up a public/private key pair to use for authentication instead of a password, though that is not covered in this post.  For additional debugging of the SSH connection on the Mac side, refer to this StackOverflow answer by Rik Renich.  Turn off the Remote Login setting from above, then run the SSH server on the command line with /usr/sbin/sshd -d.  Note that you can increase the verbosity by adding more -d arguments, so /usr/sbin/sshd -d -d -d will get you verbosity level 3.  Turn Remote Login back on when you're done.

Jenkins Setup

The guide from Jishnu Nambiar covers most of the detail needed here.  In our case, we were using Jenkins 2.479.2.  For this, we needed to make sure Java 17 (or newer) was installed on the Mac so the Jenkins node could operate.

We also found that the safest place to set the Remote Directory for the Jenkins node was under the user directory for the new user we created earlier.  In our case, we made a user called "Jenkins Node", and we set the Remote Directory to /users/jenkinsnode/jenkins.  Drive space can be at a premium, and while this is on the primary drive for the Mac, our actual Unity project is on a separate drive.  We can still access it there for remote builds while keeping the Remote Directory on the primary drive.

When using a separate location for the actual project, make sure that the full path to the location does not have any spaces.  (We had a location at /Volumes/Work Drive/Jenkins, and that failed until we renamed the volume itself to WorkDrive.)  Additionally, make sure the directory that will host the files/directories in use has full permissions for the Jenkins user (chown -R <user name> . and chmod -R 755 .).

Edit: Our extra space for the project is on an external USB drive.  We have to fully log in to at least one user on the Mac in order for the drive to be mounted.  Just turning on the Mac and running the Jenkins build wouldn't work for us because the volume we wanted wasn't mounted yet.

Git Setup

For our particular setup, our project was in Git for source control, and used Git LFS, and that required some additional setup to work on the Mac in the multi-user environment.  While Git itself was installed along with the Xcode tools, Git LFS was not.  We needed to install it separately and make it available for all users.  This was not a straightforward process.  We could not use Homebrew installs of Git LFS due to permissions issues between multiple users.

First, access the official Git LFS website to download the binary package.  Make sure to visit the website on the Mac itself, as the site will only present downloads for the OS you are currently using and hide other links.  Unzip the package somewhere (your downloads folder is fine).  Then, using a Terminal window, run the installer: sudo <unzipped location>/install.sh

From here, we need to specially link the LFS extension in with Xcode so that Jenkins can use it.  The basic technique is described in this StackOverflow post by d512.  However, due to new security restrictions by Apple, you can't make this symlink into the Xcode folder under normal circumstances.  First you need to disable System Integrity Protection, then do the symlink, and finally re-enable System Integrity Protection.  Follow this guide to disable and enable SIP.

With SIP disabled, log into your Mac and bring up a Terminal window.  Create the symlink for Git LFS: sudo ln -s /usr/local/bin/git-lfs /Application/Xcode.app/Contents/Developer/usr/libexec/git-core

With that complete, re-enable SIP.  Jenkins should now be able to run Git LFS operations, and you can continue using Git LFS with any user on the Mac.

One last step here: make sure to log in to each user on the Mac and run git lfs install from the Terminal.  This sets up some hooks in Git so that it properly sets up LFS file links when cloning repositories.  If you don't do this, Git LFS will not work properly from a newly-cloned repository.

Cabernet

Cabernet is a 19th century vampire RPG with lots of deep narrative choices.  Will you merge with vampiric society and feed on your neighbors? Or can compassion and love outweigh an eternal thirst?

We helped bring Cabernet to all consoles, including Switch, Playstation, and Xbox.  Plus GoG and EGS on top of that, really.  It's a unique game with a lot of cool story reactivity, and totally worth checking out!

Unity Logging Still Happens in Release

 We recently spent two days chasing down a major hitch, that ultimately boiled down to two things.  First, turns out, Unity doesn't necessarily disable its logging in Release, it silences it.  Second, really tall callstacks take a while to format text for.  Probably.  Best we can tell, anyway.

This was using engine version 2022.3.36f1, for reference.

This started with a major framerate hitch on one of our platforms, on the order of multiple seconds.  This was detectable on other platforms but much less substantial, and measuring it in a profiler we saw LogStringToConsole taking around 30ms in the editor, in the right situation.  That's already hitch-worthy, but it's not usually that bad and obviously much milder than on our problem platform.  So what's going on?

Initially we dismissed this, because in Release logging is disabled, right?  But as we started trying things we started being suspicious, since everything else in the area seemed benign and removing other items didn't really move the needle any.  We had assistance from QA who tested a few previous builds and verified this wasn't entirely new, but it was dependent on how long you played.  Loading into the level and testing it wasn't an issue, but leaving the game to soak for an hour was.

Eventually we got suspicious and removed all the log prints, and the hitch went away, even in Release.  Interesting.  Suggests Unity doesn't disable its log printing, sure, but why would a log print take so long even if that is the case?

Well, this particular project uses coroutines heavily.  It's built partly with Unity's visual scripting and this particular coroutine originates there.  It does its work, then at the end of execution it calls itself, and starts another iteration.  So over time you have a coroutine starting a coroutine starting a coroutine, basically forever.

Perhaps you've noticed, whenever Unity prints a log print it always includes a callstack in the log.  The editor hides this but it's present in the actual log output.  Because of the above setup, that callstack inevitably gets longer and longer, eventually being hundreds or thousands of lines long.  Why is there a gigantic hitch?  Best we can tell, because it's building that massive callstack print and formatting it.  On a slower platform, and depending on how it handles logging, this can be very bad.  Even on PC it's detectable as a few MS that maybe causes a one-frame skip.  On more controlled platforms, it's possibly going through a few layers of operations before it gets to the actual output, and who knows what those are doing

The good news is, you can actually disable logging, by setting Debug.logger.logEnabled to false.  This removed the hitch entirely, including its presence in Release builds.  So this is what we ultimately did.  But it took a long time to figure out this quirk, so here I am documenting it for the public.  Hope this saves somebody else some time.

Sprawl!

A fast-paced, hardcore retro-FPS set in a sprawling megalopolis.  Escape the walled city and take on the militarized government of the sprawl!  Sprawl is an Unreal based title, and we helped bring it to console platforms!

Sprawl is now available for Xbox and PlayStation platforms!

Encoding a Video With Alpha for Unity, Using FFMPEG

It took a surprisingly long time to track down a working combination of arguments for this, so here's one that's known working in an actual project.  This is using Unity version 2022.3.36 and FFMPEG version 7.0.x.  Allowing the asset to re-encode worked fine on all platforms we tried, including all consoles, except the SteamDeck somehow.  We ended up disabling re-encode on PC for that reason.

ffmpeg -framerate 30 -i Frames\Blink\Blink_%%02d.png -b:v 8M -c:v libvpx -pix_fmt yuva420p -metadata:s:v:0 alpha_mode="1" -auto-alt-ref 0 Videos\Blink_VidAlpha.webm