Cross-platform custom-shaped windows
This post will explain how to create transparent (non-rectangular) windows in c++ on Windows, Linux and Mac OS X. The shape of the window will be determined by an image with a transparent background. If the pixel is transparent then it will not be part of the window. It is assumed that the size of the image and the size of the window are the same.
Each operating system requires different code and will be discussed separately. Below you will find code snippets, but for a full example check this github project which creates a window with a custom shape with SFML.
In the code snippets below get_window, get_pixel_color, image_width and image_height are just placeholders. How you get these properties is up to you. In my example I used SFML, but the code below does not depend on it.
Since we want our region to be based on an image, we must adapt the region pixel by pixel. It might be slightly more performant to detect bigger rectangles in the image so that less regions have to be created, but the effort in doing so is probably larger than the speed gain.
All needed functions come from the Windows API, which means that windows.h has to be included.
Here is the code with explanations in comments:
On linux, non-rectangular windows are not available directly in the X11 library, you will need to use the X Nonrectangular Window Shape Extension Library which is part of the Xext library. On some systems the code will work, while on others it is simply not supported (the availability of the shape extension will be queried at runtime). When linking this code you will have to add “-lX11 -lXext” to the linker flags.
Except for checking the availability, the code is broadly equivalent to the windows version.
Mac OS X
Getting custom shapes on Mac is very different from the way we did it on Windows and Linux. You will not have to set any region, you can just tell the window to only be opaque where you draw on it. So by just drawing the image, the window would get the shape of the image. That may sound great, but it introduces a problem when trying to use OpenGL. When drawing with OpenGL you clear the entire screen before drawing, therefore the window would always have a rectangle shape. To work around that you must clear with a transparent color.
Mac forces us to use Objective-C or Swift (I used Objective-C here), so we can’t have the implementation in the same .cpp file as the Windows and Linux version. The code below can be placed in a .mm file. Cocoa is needed so you should also add “-framework Cocoa” to the linker flags.
Other tutorials show that in order to get a window with a custom shape you have to subclass NSWindow. I will show how to do it without inheriting from it, since you might not have access to the window class directly (e.g. when trying to change the shape of a window created with SFML).
But the above code is not enough if you want to use OpenGL. The NSOpenGLView class overrides the isOpaque function of NSView and makes it return YES while we need it to return NO. In Objective-C we can change the implementation of the function from outside the class. Add the following code somewhere and the NSOpenGLView class and all its subclasses will have this implementation.
That’s it. Now just clear the window with a fully transparent color and draw the image on it.