CUDA stands for Compute Unified Device Architecture.
The term CUDA is most often associated with the CUDA software. The CUDA software stack consists of:
CUDA hardware driver
CUDA API and its runtime: The CUDA API is an extension of the C programming language that adds the ability to specify thread-level parallelism in C and also to specify GPU device specific operations (like moving data between the CPU and the GPU).
Mathematical libraries that have been optimized to run using CUDA.
The CUDA software comes with the software driver, the CUDA toolkit (compiler, debugger, profiler), and the CUDA SDK (CUDA code samples).
More information on CUDA is available at: http://www.nvidia.com/cuda
For general discussion of setting up development environments, algorithms, optimizations and so on, please visit the DevZone forums.