In modulated crystals short-range translational order is lost and the atomic structure cannot be defined by the contents of a single small unit cell. The wave of disorder is described by a modulation function, which restores long-range periodicity. If the modulation period divided by the unit cell translation is a rational number, then the modulation is commensurate, and can be described in an expanded unit cell. Otherwise it is incommensurate. The diffraction pattern of a modulated structure contains strong main reflections from the basic unit cell, surrounded by weaker satellites from the modulation wave. Modulated structures are rare in protein crystallography.
Stress factors induce in plants the expression of Pathogenesis-Related (PR) proteins, divided into 17 classes. PR proteins of class 10 (PR-10) are well studied structurally but their biological function is unclear with an implication in phytohormone binding. PR-10/hormone complexes are studied using fluorescent probes such as ANS (8-anilino-1-naphthalene sulfonate). We crystallized Hyp-1, a PR-10 protein from St John’s wort, in complex with ANS. Solution of the apparent P4122 crystal structure was impossible by standard molecular replacement because of evident tetartohedral twinning and a bizarre modulation of reflection intensities with l periodicity of 7. The structure was solved using Phaser and data expanded to P1 symmetry. Ultimately, the structure turned out to have C2 symmetry with 28 independent protein molecules, arranged in dimers around a non-crystallographic (NCS) 21 screw along c with a pitch of ~1/7. The seven-fold repetition along c is indicative of a commensurate modulated structure: the NCS copies are similar but not identical. For instance, the consecutive Hyp-1 molecules bind a varying number (0-3) of the ligand molecules. The structure has been successfully refined to R=22.2% using conventional methods, i.e. with unit cell expanded to encompass the entire commensurate modulation period.