Currently approved epigenetic drugs target mainly metal-dependent histone deacetylases (HDACs), which are key epigenetic regulators. Yet, cross-reactivity of these drugs for the structurally similar but functionally different HDAC isozymes hampers their broad usage in clinical settings. Selective inhibitors targeting single HDAC isozymes are being developed, but our precise understanding in molecular terms of their selectivity remains sparse. PCI-34051 and NCC-149 were among the first inhibitors displaying isozyme-specificity for their target, HDAC8. In this report, we show numerous structures of HDAC8-inhibitor complexes solved by X-ray crystallography. These structures reveal how HDAC8-selective inhibitors build their selectivity on specific interactions, notably with the HDAC8 active site catalytic tyrosine, but also through contacts with the HDAC8 L6 loop that forms, together with the L1 loop, a HDAC8-specific pocket. These interactions are enabled by the specific size and conformation of the HDAC8 L1 and L6 loops, which leave the catalytic tyrosine uncovered, and by the constrained L-shape of HDAC8-selective inhibitors. Collectively, our results highlight the importance of HDAC active site loops and architecture, and pave the way for the design of next-generation selective HDAC inhibitors.