In mathematics, specifically functional analysis, a trace-class operator is a linear operator for which a trace may be defined, such that the trace is a finite number independent of the choice of basis used to compute the trace. This trace of trace-class operators generalizes the trace of matrices studied in linear algebra. All trace-class operators are compact operators.
In quantum mechanics, mixed states are described by density matrices, which are certain trace class operators.
Trace-class operators are essentially the same as nuclear operators, though many authors reserve the term "trace-class operator" for the special case of nuclear operators on Hilbert spaces and use the term "nuclear operator" in more general topological vector spaces (such as Banach spaces).
Note that the trace operator studied in partial differential equations is an unrelated concept.