Welcome! This guide aims to demystify how to create those "magic piano" visuals using Keysight's 3D camera features. This is a large topic and guide with no clear "this is the best way of doing things", but the following is how I personally align augmented-reality visuals. While subheadings do allow you to jump around this guide, I strongly recommend reading everything in order and not skipping any steps!

Requirements

In order to perform AR alignment, you will need:

• To be in Advanced menu mode
• A stable piano and camera, so things stay aligned
• An image of your camera angle at the target aspect ratio
• Semi-transparency (Mask mode if streaming, Video backgrounds if making videos)

• (Optional) A game controller

• Patience! This is one of the most difficult things to do with Keysight!

Process

This guide will use a stock Keysight installation in Advanced menu mode.

Deciding on a camera angle

Camera angle is important to getting a convincing effect. If the camera angle is too shallow, your hands may end up inside the visualisation area. If this happens, the visualisation will appear on top of your hands, and break the illusion! Typically an "over the shoulder" angle from off to one side is a good idea, just take a look at how other creators have done it. Here is the angle I will be using to demonstrate in this guide:

I cannot stress enough how important it is to have your camera and piano nice and stable too! AR alignment often takes a lot of time to get perfect, especially if you're not practiced at it, and any time you nudge your camera or piano you will have to go through most of the alignment process again and lose a lot of time.

Setting an alignment image

First, take a screenshot of your camera angle. If it's a webcam hosted in OBS, a good method of doing this is to right-click the canvas and then select "full-screen preview" on a monitor that is the correct aspect ratio.

Once you have your image:

• Head into Core > Visual layout
• Switch to Manual layout mode
• Scroll down to Augmented reality alignment section
• Turn on Enable alignment image overlay
• Click the + icon
• Add your image

Preparing a preset to use in alignment

We will be using the black keys in Keysight as a reference point against our alignment image, so the next step is highlighting these black keys.

• Create a new preset, call it "Alignment reference" or similar
• Widgets > Light bars, disable all Light Bars using the tab's toggle
• Under Scene > Frame, enable Frame uses custom material and set it to Transparent mode with 0.00 Base opacity value
• Scene > Piano > Black key material, set Self-emissive colour's Emissive strength to 1.00, and colour to pure red
• Scene > Backdrop > Material, set Self-emissive colour's Emissive strength to 1.00, and colour to pure blue

Now you should have something that looks like this, with clear black key notes and active area, and no shadows on the piano obscuring things:

Setting up the Keysight spatial layout

Next, we need to decide on the approximate angle of Keysight's Backdrop and piano in relation to each other. The goal is to mirror your physical layout, so that the Keysight piano sits exactly on top of your physical keyboard, and the active area is exactly where you want notes to appear.

Backdrop angle is determined by Core > Visual layout, Camera position section, Origin Y (up/down) with 90.0 giving a "flat" Backdrop and 0.0 giving "vertical". For this example, I am going to use 45.0 for an angled view. After changing the value, you will likely need to re-acquire the active area by looking up in the first-person fly camera mode (by hitting the Enter fly cam mode button, or F while the menu is closed. Controls shown in top right of the screen while active. This is where the controller comes in handy!).

Next, the Keysight piano's angle and position relative to the Backdrop is determined by Core > Visual layout, Keybed position section. Enable Disconnect piano keyboard from backdrop to allow editing.

The first step is to counteract any Backdrop rotation given by Origin Y (up/down). In this example, I need to apply -45.0:

Next, using the fly camera to loosely position the camera in the right spot, we want to adjust the Local additional vertical/height positions so that the Backdrop is displaced from the piano by roughly the distance seen in your physical piano. These values are given in centimeters, if you want to measure things rather than just eyeball it. I am choosing to align my backdrop to the green line, and so I want my piano displaced downwards a little.

Finding the correct field-of-view

Use the fly camera (F while outside the menu) to loosely align the keyboard with the physical keyboard in your alignment image. The goal here is to be at the correct height above the piano such that it mirrors the physical camera location. The angle of piano keys in Keysight is a good way of judging this; if they are angled differently between Keysight and the physical view, you're probably at the wrong height (although an incorrect field of view can also result in incorrect key angling!).

After loosely aligning, fine tune to get the outermost black keys matched up between Keysight and the physical view.

Now, purely by accident, the field-of-view (FoV) in Keysight pretty much a perfect match of the camera's field of view in this example (hence the black keys lining up perfectly!). This almost never happens, and you usually end up with something that looks more like this:

In the above image, the FoV in Keysight is too wide. You can deduce this based on the excessive perspective warping; with closer black keys being far too large and angled while the distant black keys are the correct size. Alternatively, you may get something like this:

In the above image, the FoV in Keysight is too narrow. You can deduce this by the spacing of the closer black keys being too narrow, and the angle being wrong. Try out some different FoVs until you get one that looks pretty close, like this:

Once you're close to getting the correct FoV, adjust by increments of 2.0 either direction to begin with, making these increments smaller as you hone in on the "true" FoV. The goal is to have all black key spacing perfectly match across the whole keyboard, like this:

In the above image, I went down to 0.25 increments of FoV to get it this accurate. Even still, there are slight deviations! But being within 1.0 degree of "perfect" is definitely good enough for 99% of situations.

Final alignment

Finally, we only need to slightly tweak the angle to move the Backdrop into the final position! At this point, we can ignore the piano alignment, as this was purely used for finding the correct FoV and we will be hiding the piano in any "real" presets.

Polish

Things are now aligned! But we're not quite finished yet. There's a lot of tidying up to do.

Backdrop height

You likely will want notes to scroll off-screen. If the top of the active area is visible, notes will be terminating on-screen and look pretty bad (unless applying some kind of opacity fade in OBS or similar). Use Core > Visual layout, Active area height and increase/decrease it such that your blue highlighted Backdrop only just goes offscreen over the whole width:

Preparing a preset for use in augmented reality

Taking the default Quicksilver preset as an example, you will likely want to adjust the following things:

• Disable Effects > Keypresses
• Disable Effects > Impact lights if these only affect the piano, as is usual (saves on performance)
• Disable Scene > Frame
• Disable Scene > Piano
• Under Scene > Backdrop > Behaviour, enable extend backdrop bounds horizontally. This allows note lighting to spill out wider than the active area, and prevents awkward cutoff at the edges
• You may need to adjust the Z-height layering of objects like Light Bars, or extend the width of Light Bars. This is highly preset-dependent
• You may need to remove reflections from the Backdrop material, as these can awkwardly cut off along the bottom edge

Foreground elements

It can make a big difference to have something intentionally "on top of" Keysight obscuring the visuals. This is achieved through alpha masking the Keysight visuals in OBS or your video editor with a mask created from your alignment image. In this example, the tablet makes for a good foreground element, so we're going to carefully select that in an image editor (I am using GIMP➚ here, as it is free and open source):

The goal is to assign pure black to any areas where Keysight should be obscured by the foreground elements, and pure white elsewhere:

Then apply this mask to the Keysight source, using the colour as the alpha for Keysight. In OBS, this should occur after Mask Mode, and using an Image Mask/Blend filter. The final result should look like this, note how the bottom edge notes are now "behind" the tablet and the whole thing feels a bit more "real" as a result:

Anti-aliasing

By default, Keysight has anti-aliasing disabled in order to make particles look sharper. In an angled augmented reality application, you likely want anti-aliasing on Ultra to soften things. This can be found under Graphics > Quality > Anti-aliasing quality.

Troubleshooting

Of all the common Keysight implementations, augmented-reality is the most challenging! Don't panic if you have some difficulty, and consider joining the Discord➚ to ask specific questions!

Creating a backup

I strongly advise you back up your aligned angle once you have it! Things get nudged, or you may accidentally default your Keysight settings, or some other disaster occurs and you definitely do not want to go through the full alignment process more than once.

To create a backup of the Keysight angle:

• Core > Visual layout, click the "Copy" icon to copy your layout settings
• Core > Visual layout, switch from Use global settings to Use local settings
• Core > Visual layout, click the "Paste" icon to paste your layout settings

This has now baked those layout settings onto the preset itself, allowing you to export the preset (using the bottom middle icon on the preset tab) to somewhere on your hard drive and keep a backup of your alignment! If you ever need to restore this backup, simply import the preset and copy/paste the local core settings from the alignment preset onto the global core settings (either in the same preset after switching it, or onto any other preset that uses global settings).

It may also be beneficial to "back up" the position of the physical camera too. Cameras get nudged, but it is sometimes feasible to nudge it back into position if you have an alignment guide as a reference. The way I like to do this is to open up an image editor and add sharp, bright lines to obvious reference points in the scene, and then save out just those lines as a transparent image. I then use this image in OBS to confirm that my camera angle still matches up with the angle that I performed Keysight AR alignment to:

Z-axis of effects

With angled views, the 3D nature of Keysight is exposed. Be very cautious with aligning non-Backdrop elements to your desired alignment point, as they commonly have some Z-height offset to them! Note objects have a 0.1cm height offset by default, and many Light Bars have a 1.25cm offset. You may need to substantially alter the default Z-height layering of tons of effects to get your augmented-reality view looking good, as the defaults in Keysight are intended for a 2D view.

Vertical video alignment

Aligning to a vertical view can cause some headaches as you need the Keysight window to also be vertical and 9:16 for the alignment image to be unstretched. Unreal Engine does not allow Keysight to resize to a resolution larger than your available monitor resolution.

• If you have a 1920x1080 monitor: use 603x1072
• If you have a 2560x1440 monitor: use 810x1440

However, going into such a 9:16 aspect ratio will break Keysight's menus. As such, use the shortcut of alt + enter to toggle between Borderless windowed and Windowed modes. This allows you to rapidly switch between broken-menu-9:16 mode and working-menu-16:9 modes without needing to actually hit any menu buttons in the process.

Removing planar shadowing

If you have planar shadows, particularly across the top of your Backdrop: these are granted by Light Bars and must be disabled or adjusted under Widgets > Light bars.

Using the Randomizer

Under a stock setup: using the Randomizer with this kind of 3D view will turn a bunch of unwanted effects back on and completely ruin the visuals. To combat this, you will need to change the internal "default preset" that the Randomizer uses as a baseline to randomize from.

• Make a new preset
• Adjust it according to the steps above to get it looking good in AR
• Consider only removing the default Light Bars rather than fully disabling them, so the Randomizer can generate a Light Bar
• Go into System > Defaults calibration and click Set current preset as default
• Under Randomizer, disable the Keypress and Impact light tabs
• Save the Randomizer and System Settings using the bottom-right buttons to save this

Now, the Randomizer pulls our AR-ready default preset and does not re-enable Keypresses or Impact lights, and functions as you would hope. However, be warned that Z-height of effects is derived from templates, and so randomizer-generated presets will likely have undesirable Z-height layout of effects like Light Bars.