One way that stars are categorized is by temperature. Since the temperature of a star can determine its visual color, this category scheme is known as spectral type. The main categories of spectral type are M, K, G, F, A, B, and O. The coolest stars (red dwarfs) being M, and the hottest stars being O. Our own Sun is a G star.
Red dwarfs are small stars in terms of their mass. Large red dwarfs have a mass have about half the mass of our Sun, but they can be as small as 0.075 solar masses (a bit more than 78 Jupiter masses). Below that, and a star simply doesn’t have enough mass to fuse hydrogen in its core. Since hydrogen is the most abundant element, and its fusion is the main source of energy for a star, this means an object with less than about 78 Jupiter masses isn’t able to produce energy the way main-sequence stars do. You might think, then, that this marks a clear line of demarcation between planets and stars. Anything above about 78 Jupiter masses is a star, and anything below that mass is a planet. But it turns out things aren’t quite so clear.
A star with a mass less than 78 Jupiters is often referred to as a brown dwarf. They are sometimes referred to as “failed stars”, but this is a bit of a misnomer. While brown dwarfs are too small to fuse hydrogen, they still can fuse other elements. If a brown dwarf is larger than 65 Jupiter masses, it can fuse lithium. If it is larger than about 13 Jupiter masses it can fuse deuterium (which is a less common variation of hydrogen that has a proton and a neutron, rather than just a proton).
The largest brown dwarfs can have surface temperatures as high as 2,800 K. While this is about half the surface temperature of the Sun (5,800 K), it would still appear very star-like to the casual observer. In fact hot brown dwarfs can be difficult to distinguish from cool red dwarfs. One way to tell them apart is to look for lithium in their atmosphere. Red dwarfs, because they can fuse hydrogen, also fuse most of their lithium early on (a process known as lithium depletion). Brown dwarfs, on the other hand, are never active enough to fuse most of their lithium, so it is still present in their atmospheres.
Because brown dwarfs can’t fuse common hydrogen, they never produce enough energy to keep their temperatures high. So as brown dwarfs age, they cool down. Since stars are categorized by their temperatures, this means brown dwarfs change category as they age.
The warmest brown dwarfs can be in the cool end of the M category, but the first true brown dwarf category are L dwarfs. These have surface temperatures of about 1,300 K – 2,000 K. They are cool enough that lithium and other alkali metals are present in their atmosphere. Below the L category are T dwarfs, with surface temperatures of about 700 K – 1,300 K. These are brightest in the near infrared, and are cool enough for methane and other molecules to build up in their atmosphere.
But even the T class dwarfs are not the coolest stars. Recently the WISE infrared telescope discovered six very cool brown dwarf stars. Since they all have temperatures less than 600 K, they are placed in a new spectral type known as Y dwarfs. The coolest of these has an atmospheric temperature of only 300 K, or about room temperature.
You might think a star with an atmospheric temperature you could comfortably lounge in should hardly be called a star, and there are astronomers who would agree with you. They certainly would appear much more like Jupiter than the Sun.
Because of their extremely low temperature, the Y dwarf spectral type is a bit controversial. But these Y-dwarf
stars do appear to be larger than the minimum 13 Jupiter masses necessary to fuse deuterium, and if we use fusion as a defining character they would technically be stars. That would make them the coolest stars.