Designing interior courtyards will generally follow the same rules for designing small gardens. The real major difference is in paying more attention to vertical space - the walls that surround courtyards. Some small gardens do require this attention but almost all courtyards have this extra consideration.
Generally, the first thing that comes to mind for decorating walls and vertical space is to cover them in climbing vines. While this has a lot of good character, it only decorates the existing boundary without creating a depth or illusion of boundary.
In most cases I've found that creating an independent source of height in front of courtyard walls and /or in corners can create a 3d effect much like that found in paintings. In a sense, it makes the independent element a focal point while using the decorated walls as a backdrop or framework. Framing an element this way helps create depth of boundary behind that element.
Some good elements to use are small trees with tall trunks, tall pots, pots on stands, groups of pots, trellis works, and decor. Creating beds next to walls, filling them with the same vines as the walls, and placing a few tall specimen plants or trees also has a nice dramatic framework effect with a lot of depth.
The color of your walls also makes a big difference in the depth, atmosphere, and mood of the courtyard garden. Bright and light colors have a closer closed in feel while darker colors add more depth and a sense of more space and distance.
Wall decor, wrought iron wall decor, clay ornaments, and wall fountains are also some good elements for effectively using vertical space in small closed in gardens. When using elements like this, pay special attention to the principles of unity, simplicity, and repetition. Stay consistent. Using more and varied objects can start to look very cluttered.
Small garden design and designing for courtyards needs to pay closer attention to detail rather than filling up space. Using more is not always better. And sticking to just a few elements a few basic design principles will create a much classier clean atmosphere.