Compiling C++ quickly in Visual Studio

Ensuring fast C++ build speed requires some effort--

Modules should improve the situation, but that is some years off(C++17 if we are lucky).

Here is what has worked for me(using Visual Studio)
This is intended for a rapid iteration build, a build you intend to ship would not use these settings.

• Enable /MP Multiprocessor builds. 
• Disable Global optimizations
• Incremental Linking (/INCREMENTAL) and Use Library Dependency Inputs(Yes)
• Add /Zc:inline  to command line of compiler
• Linker->Optimizations: References(No), COMDAT Folding(No)
• Pre-Compiled Headers(PCH). Put rarely modified files in the PCH, but include everything you can possible include. Test the compilation times of files that use the PCH. You can enable this in Visual Studio(*2). Try to get it as low as possible.
• Limit the headers you include, especially avoid including headers within headers
• Forward declare everything that can be, even types passed by value can be forward declared
• Do not create "registry files". *example: a file containing an enum with an entry for each type of component in your engine. If you do this, you will need to add an entry to this file each time you create a new component, which will result in a large and often triggered rebuild cycle.
• If you find yourself calling from Module A to an object in Module B, but only doing so once, think about wrapping that access in a C function call. The C function can be placed in Module B, then either declared extern within Module A, or a separate header can be created if it is likely that this C function will be called multiple times. This reduces coupling between files. Changes to A will not trigger a recompilation of B.
• /PDBCOMPRESS slows the linker down, should probably not use this
• Use type erasure to reduce coupling & generate less code. For none performance critical paths, a single type erased path can be preferable to a template path per type.  
• Focus on optimizing the build speeds of any commonly used templates. There are multiple things you can do here.
• Some templates can be simple wrappers around a none templated type or function, which is itself implemented in a .cpp file, and thus not inline.
• Use variadic templates to replace old style overloads based on # of arguments.
• If there exists a finite set of types that the template can be instantiated with, try explicit template instantiation.

Templates are commonly accused of killing build times.
This is not very accurate, the real culprit is headers. Since templates exist in headers it gives the false impression templates are at fault.
 Once you have solved the exponential build time issue caused by headers, templates have a only a small effect on build times.

*2: To enable build timings: 

Tools -> Options -> Projects and solutions ->VC++ Project Settings->Build timing->yes

**Tools -> Options -> Projects and Solutions -> Build and Run and set the MSBuild project build output verbosity to "Normal"** 

Carving

Various voxel landscapes I made recently.

The carving tool is still pretty basic so I can't do anything impressive yet.

You basically can select from 5 shapes + 4 blend ops.  

But it works in real time so I can fly about and apply these ops and see the result immediately.

Chile & Argentina

Here is a link to my friends blog detailing our three month trek in South America.
Photos include famous stuff like Torres Del Pine, Tierra Del Fuego, and Fitz Roy.

Probably the most memorable bits:
-trekking through lava flows created by the 2011 eruption of Volcan Pueyhue;  40 mph winds filled with ash, visibility was almost nothing, we would stumble to the edge of chasm, gaze down, and have no idea how deep it was:) Had to use GPS/compass to navigate our way.
-living in a tiny hut on a glacier for five days crammed with 12  people, waiting for a storm to pass--

Lots of nice photos on his blog

Snowfall in Tierra Del Fuego

Poking my way across a glacier

color

Coloring algorithm is dead simple.

Eventually I want to incorporate the elevation & slope.  

It is using a (mostly) physically based lighting model, but the input materials were typed at random by me, and don't match any real world data.

Current input signals to lighting: red, green, blue, roughness, spec, ambient, shadow

bad colors

A voxel hill, so shiny

dug this out

*The colors were generated by randomly entering values between 0-255 into an array.

Sculpting Voxels In Monocrome

 Modifying the world at run time is now possible. The primary difficulty was minimizing which parts of the world needed to be rebuilt.

Yeah, I made this- what is it? No idea.

Giant letters, hoarding tiny balls

 In the following image, each of the balls is actually a separate voxel structure carved into the form of a sphere. They each have a spherical collision shape associated with them, and are able to move about within the larger world voxel shape.

Voxel balls barreling down the mountains side

 Unfortunately for this project I am heading down to South American for a few months and won't be working on it during that time. Ciao.

Shadows & Progress

• 12x faster world generation speed
• Added a soft shadow pass





Sun at shallow angle for long shadows





The space boulder from afar.  

• 
  

Left:Shadows casting across the entire rock/planet/placeholder I'm using

Cavern shadow!

let there be noise--

  Was on a trip for about a week so didn't get much done.  Added basic  noise support.  The noise is applied to the sphere here.

Some parts have noise, some parts don't.
Formula:(min (add sphere noise) (repeat donut (repeat hexigon (repeat sphere))))

Everything has noise here
Formula:(add (add sphere noise) (repeat donut (repeat hexigon (repeat sphere))))

Formula:(min (add sphere noiseRMFBillow) (repeat donut (repeat hexigon (repeat sphere))))

Formula:(min (add sphere noiseRMF) (repeat donut (repeat hexigon (repeat sphere))))

Formula:(sub (add sphere noiseRMF) (repeat donut (repeat hexigon (repeat sphere))))

Graphics are pretty bland right now, but I've got more important things to work on for at least a few months. Might touch up the graphics early next year.

LuaJIT info & links

 LuaJIT

• Size of coroutine: 400 bytes 
• Each VM requires ~10k physical, 128k virtual memory
• JIT disabled on iOS because Apple doesn't support run time code generation
• JIT disabled on Windows 8 for ARM for the same reason
• LuaJIT interpreter(no JIT) is ~3x faster than standard Lua
• LuaJIT with JIT enabled is ~20x faster than standard Lua
• Lua C API  prevents JIT compilation
• LuaJIT FFI is much faster if compiled, but if not compiled(like on iOS), it is slower than the Lua C API
• LuaJIT FFI cannot be safetly sandboxed, do not use for user scripts(Mike Pall recommends process isolation)
• LuaJIT GC considerations by Mike Pall
• 1 to 2GB/ memory limit(depending on OS and x86/64) for current LuaJIT, extended to 2-4GB whenever the new GC arrives , but you wouldn't ever want to allocate that much memory
• All LuaJIT allocations must be within first 2GB of address space
• LuaJIT 3.0 will have a brand new GC, optimized for real time applications
• Tracing compiler discussion with Mike Pall(LuaJIT) and Brenden Eich(Javascript), worth reading the entire thing
• Design of LuaJIT: optimization, register allocation
• Lua 5.2 __gc metamethod not support for tables yet in LuaJIT, can use newproxy, an undocumented Lua function, to work around this
• LuaJIT bytecode 40% smaller than standard Lua

C++ Variadic Templates

 I've been using Visual Studio 2013 RC which supports C++ 11 variadic templates.

 Here are a few things I've noticed.

Good

• Compile time reduced. Replacing my N duplicates of a template with a single instance means less work for the compiler
• Less code. Is some cases *way* less code. This is great! Can replace manually duplicated templates with 1 instance, can also replace Boost Preprocessor generated templates with variadics

Bad

• If you step into a variadic function in the debugger it does not always allow you to see what values were passed into the function.  I did see it work once, so perhaps it is just buggy/incomplete and the final release for VS 2013 will fix this.
• The following code does not compile, to work around it, as far as I can tell, you must resort to C++ 03 code(boilerplate). You couldn't instantiate this type because it doesn't map to a struct/class, but it would make it easier to pass type information around.  

edit: Looks like there is a proposal to fix my last point- N3728

CDT

 So I walked into Canada on September 5th.  To see all the places from Mexico to Canada along the Continental Divide was pretty amazing.

In the New Mexico desert always on the hunt for the next water source(cow troughs mostly)
In the Colorado San Juan mountains struggling to make 20 miles per day in the snow and altitude.
In Wyoming the Wind River Range is possible the best mountains I've seen, and I've seen many
In Montana crushing out its 900 miles with my friends SweetAs and Aquaman.
And then walking into Canada with Balls and Sunshine, whose journal you can read if interested

My first Grizzly bear encounter was about 10 miles from the monument that marks the US/Canada border. Only saw three bears on this trail, half as many as on the PCT.

 Here are a few pictures, nothing can really capture a 4.5 month/3000 mile hike, but this will have to do

Cirque of the Towers in the Wind River Range

SweetAs and I in the Wind River Range

Climbing up to the top of the Divide

Fetching water from a cow tank in New Mexico

Hiking with Breeze in Burglar across the wasteland around Lordsburg, NM

Grapple and Thunder along the CDT in southern Colorado

 Another hiker this year kept a detailed blog of her hike. Her name is Wired and she was a few days behind me for most of the trail(until Montana).  You can find her blog here.  It has tons of pictures and is high quality.

CDT

 Hiking the Contential Divide Trail(CDT) for the next 4-5 months. It is a 3,000 mile trail running from the US/Mexico Border in New Mexico to the US/Canada border in Montana.

 I'm starting April 20th at Crazy Cook, NM.  Will walk across New Mexico, Colorado, Wyoming, and Montana, and some of Idaho.

Old post by Tim Sweeny regarding Garbage Collection

So UClass/UFunction/UProperty(which still exist in Unreal) etc. appear to date to at least 1999

Tim Sweeny 1999

Sun Shafts & SSAO

 Added screen space sun shafts, based on this article from GPU Gems 3.

Also added basic SSAO(screen space ambient occlusion).  Right now it is just using the depth buffer.  For performance I will switch this over to using a downsampled depth buffer at some point in the future.  There are also some alternative methods that take into account surface normals that tend to generate more accurate results, so I might add one of those methods later.

Ambient Occlusion

FXAA

 Added FXAA, aka lets blur stuff

Atmospheric Scattering

       Atmospheric scattering based on Eric Bruneton's Precomputed Atmospheric Scattering.

 I really like the look this algorithm gives, I'm only part way through implementing it so not everything is working, or at least as well as I'd like.

 His algorithm produces three colors, inscatter color, ground color, and sun color.  Although Bruton's sun color looks good in his screen shots, in the demo it looks rather bad; I believe the demo is an incomplete version.

 Part of the algorithm requires the distance from the viewer, here I am reconstructing based on the depth buffer, and to get this working I went ahead and implemented a proper deferred renderer, so yeah!

  Oh, and the first time I ran the atmospheric code this is what my planet looked like(precomputed table wasn't right )....

Deferred Texturing ?

  One method I tried, that didn't end up working very well, was deferred texturing for the terrain.

The terrain texture coordinates are based on its world coordinates and normal.  Using the GBuffer normal + depth buffer to reconstruct world position gave me what I needed.

 And while this worked and produced identical texture coordinates, the gradients were not always correct, resulting in some nasty looking aliasing anywhere that the depth buffer contained a large difference between adjacent pixels(because originally they were separate objects).

 My understanding is that the graphics card works on 2x2 pixels--

AB
CD

So the world coordinates are calculated in the four pixels, the difference between the adjacent pixels world coordinates is used as the derivative for MIP calculation.

But if pixel A was from a completely different mesh than pixel C, and 1000 meters closer to the camera, it ends up with really large derivatives and selects the lowest MIP--this is not what you want!

 So everything ends up with a blocky/pixelated outline that shimmers and looks just awful.

 If you passed down the derivative information perhaps you could work around it, but that is quite a large amount of extra data, and I didn't want to go down that route.

Terrain Video

 I made a short little video flying around the terrain, 1080p if you click on on the word youtube.

Terrain Blending

NOTE: the methods in this post are old & moronic, don't use them:)

 Blending:

 I've updated the terrain so that it blends seamlessly between different LOD's.  Previously it would just switch from one LOD to the next without any attempt to blend, which looked pretty bad, especially if running the terrain at a fairly low detail setting.

 Textures: 

  For now just using some random textures off the internet.  Texturing is being applied using triplanar texturing.  Currently it is just a global set of 3 textures, but eventually I plan to make it store per vertex a texture ID, so that the terrain can be, at least to some degree, painted upon.

Seamless blending & basic texturing

 
  Some blending details...


       Typically terrain blending is done via vertex morphing & manual texturing blending, at least this is how most height field based algorithms work.
   I am using voxels though.  Rendered as chunks of triangles, essentially each LOD's meshes are unrelated to the next LOD's meshes.  There is no real obvious way to morph between these meshes since they have nothing in common.
     What I have seen done before is alpha blending between LOD's.  Nvidia did this in their GPU terrain demo.  I plan to use a deferred renderer, which generally runs counter to alpha blending, so I've modified this idea somewhat.  Currently what I am doing is rendering high detail terrain into one set of buffers, and low detail/parent terrain into another. In post I blend them together based on a few different things.  The overhead for this is actually surprisingly small, and eventually I will be able to blend the data together prior to lighting being applied, essentially dumping the result into the deferred gbuffer.


 and cracks..
          Are gone, mostly.  I added a check so that if no data is found in the high details, it tries to fill it from the low detail, this seems to fix most cracks. I will probably expand on this to remove all cracks at some point in the future, but it removes most already.

Blue represents any location that was hole filled or crack patched, red represents places where fixes failed.  (The sky is red, which is good since it is shouldn't be patched).

Components system + Flow Graph

 Like lots of other games, my game uses a component system.

This system is designed to allow for improved cache usage and easy parallelization.
It is loosely based on this article published by Insomniacs.

For cache friendly behavior all components of a given type live in contiguous memory, and are all updated together.

To specify attributes for components I have a Lua file,  which for each component type indicates, among other things, how they should update(parallel, serial), and which other component types they depend on.

 I use TBB Flow Graph to express dependencies, any component that has a dependency on another being updated prior to it, just lists that component as a dependency in the Lua file.


For instance this is the declaration of my Occlusion Rasterizer component in Lua.

reg("occlusion_rasterizer",    no_inteface,   TM.MULTI,  0.0,  3, {"commands"}) 

The parameters in order are:

name      - name in C++, this matches them up
interface - specifies which interface, if any, the component implements, if Occlusion Rasterizer had implemented an interface this would be the name of that component.  This allows for things like finding all components that implement a given interface.
parallel      - either SINGLE or MULTI, SINGLE means each component of this type is updated sequencially, MULTI means they are all updated in parallel
time delay - how long between updates to this type, so if you want a given type to only update  4 times per second make this value 0.25 etc
count      - how many components to reserve, this allocates a contiguous block of memory. It will grow as needed, but this allows for the equivalent of a vector.reserve()
dependencies - list of any components that must update prior to this component, TBB flow graph guarantees they will finish updating prior to this component updating. Occlusion Rasterizer depends on a command list being generated, so it lists "commands" as a dependency.

 Each component type derives from a base component type in C++, currently this adds 12 bytes of overhead to each instance in 32 bit builds, and 16 bytes in 64 bit builds.

The base component contains this:
vtable ptr -  4/8 bytes. Removing this is possible but makes the system somewhat clumsier to use.
chain index  - 4 bytes, which chain it is on, chains are a series of components
type            - 2 bytes, each component type has a unique ID
offset          - 2 bytes,  index into the pool containing components of its type


 I've been pretty happy with this system, my components are small self contained little objects,  they are cache friendly, and can be allocated and destroyed in large numbers. Adjusting dependencies as new types are added is also a very simple since no recompilation is needed, just a chance to a Lua script file.


 A major advantage of component systems, which I think does not get stated clearly enough, is how much glue code they save you from writing.  Having to create classical C++ game objects and populate them with appropriate member data, worry about all the fragile and arbitrary hierarchies, and then writing all the systems that control them is a huge amount of unnecessary code.