Games engines are vital, but often overlooked by your average gamer. Games engines are the code that makes things work. They keep a track of health points, keep a track of collisions, send 3d models to the rendering controller so they show up on screen. Basically Engines are code.
Games engines can essentially be broken down into different areas, for example; Graphics rendering, Physics calculation, Networking controller, etc. These different parts all work together in conjunction with each other to produce the games we play. Depending on the skill and knowledge of the programmer that designs them, different games engines vary in quality.
The way in which a games engines creates and displays its 3d matrix varies, but generally is either considered 'subtractive' or 'additive'. The difference between these two rendering methods is quite simple. In games such as 'Half-Life 2' there is a 'void' in which you put objects into. The void is infinite, and in such an environment, the designers need to close the level off from the void. (this is because lighting information cannot be calculated in the void, due to its infinite distance, if the engine tried to calculate the distance traveled by a 'light ray' into the void it would never stop calculating, and the game would hang).
In subtractive environments, such as the 'Unreal engine', rather than having an infinite 'void' there is an infinite cube. Rather than the developer placing objects into the void and closing it off, the developer must first carve out an area in which to place objects. The benefits of this are that the developer does not need to spend important time checking the level to ensure there are no 'leaks'. However, the developer does need to keep carving out space to use, so it is two sided sword.
Developers have an option whether to buy a new games engine, or write their own. There are pros and cons of both.
Games engines are big, fucking big, not to mention complicated. Its very easy to write a poor game engine. It is also very time consuming to write a new games engine. In today's industry where 'time is money', this can often be seen as a risky move. However, if a game being developed, requires features that are not well supported in a current games engine developers can be forced to write or modify.
It is also worth noting that while writing new games engines may be quite rare in the current state of the industry, many developers are building upon old engines that they have used for previous games. For example Bohemia Interactive have been developing their 'posideon' engine for three game generations now, improving upon it, rather than writing a new engine.
However, some developers choose just to purchase the license for an engine that has already been written by either another studio, or a private company. Many engines are commercially avaliable, for example the 'Torque' engine can be purchased for a very small price. Other more well known engines such as the 'Unreal' engine are also commercially available, however, for a much higher price.
As time goes on, so do engines, and they must improve along the way. As new games demand new features, the new engines must provide them. In current engines there is a definite exchange between playing space or detail. For example, 'Operation Flashpoint 2' being developed by Codemasters requires large playing areas, while still trying to remain aesthetically pleasing to its audience. it will be interesting to see how Codemasters overcome this challenge with their new engine.
Online games are starting to look 'old', but without improved networking controllers, the developers won't be able to add newer features and higher levels of detail. (the more 'packets' an engine needs to send to the server, and then receive, the slower the latency, online games try and optimize this as much as possible).
There are many more small challenges that confront engines, and it is the resolution of these challenges that keeps the industry progressing graphically, and in many other ways.