Cloudification of the raw cryo-EM data and metadata using iRODS

Radek Veverka1, Radek Furmanek2, Jan Martinovic2, Martin Golasowski2, Jiri Novacek1

1CEITEC, Masaryk University, Kamenice 5, 62500 Brno, Czech Republic

2IT4Innovations, VSB TU-Ostrava, 17. listopadu 2172/15 708 00 Ostrava-Poruba, Czech Republic

jiri.novacek@ceitec.muni.cz

Service cryo-EM facilities are responsible for acquisition of a considerable portion of the single particle and cryo-electron tomography data generated worldwide. Each cryo-EM experiment usually produces at minimum 0.5 – 2 TB of raw data that first need to be made available to the researcher (the data owner) and afterward should eventually be made publicly available with proper annotation. In addition, it is nowadays a standard practice that both single particle and cryo-ET data are pre-processed on-the-fly to gather information about data quality and fasten the downstream data analysis. The service facilities thus need to invest and maintain additional computational and storage resources apart from the electron microscopy instrumentation. We have developed a workflow that facilitates raw data management and runs the on-the-fly data analysis on remote high-performance computing (HPC) resources. Our workflow is based on an engine accessible directly from the microscope computer via a web browser, which harvests the metadata and starts the data transfer and analysis in parallel with the initiation of the data acquisition. A federated cloud solution based on iRODS was selected to carry out the data transfer to the storage close to an HPC center, where the data are submitted to Relion, CryoSparc, or Scipion pipeline. The results of the data analysis are collected to update the data acquisition parameters if necessary. Subsequently, the data are made discoverable to the owner for the subsequent data analysis or the raw data are later transferred to different storage within the iRODS zone for archival. In parallel, the metadata can be made publicly available through EUDAT.