Storage Recources
Computer Center
Storage Recources
1. Overview of different Classes of Storage
1.1 AFS
1.2 dCache
1.3 Lustre
1.4 DESYcloud
2. General Recommendations
3. Further Readings
1. Overview of different Classes of Storage
The computer center provides various types of storage recources and file systems. Please inform about them for the efficient usage.
In case of questions or problems concerning the usage of storage resources please contact uco-zn@desy.de.
1.1 AFS
AFS is a wide-area filesystem. It is optimal for holding small login scrips, eventually some code and small ntuples. Backup is possible.
DO keep scripts and small important files for backup
DO NOT have more than 65k entries in a directory
1.2 dCache
dCache is the main storage system which is entry point for all data and exchange point with the Grid world. Files in some directories are only cached on disk till written to tape. A second copy for valuable, unique data can be done on request.
DO store analysis output and large files which should stay and eventually be visible from the grid.
DO NOT put small files here.
1.3 Lustre
Lustre is a cluster file system and a scratch area for group data without backup.
DO reading and writing large files with high bandwidth.
DO NOT store too many small files.
1.4 DESYcloud
DESY Sync & Share is the cloud storage service of DESY.
DO share files with others here
DO NOT store important data only here
2. General Recommendations
- store your most valuable data on an adequate storage area with backup
- the disk storage is hosted on RAID6 arrays, but keep in mind: we have to expect an average loss of a hundreds gigabytes of such data per year due to multiple disk failures, defective controllers or cache memory, firmware bugs, and even mistakes made by the administrator.
- don't waste your storage resources and use them efficiently
- please don't waste space with backup for bulk data rewritten daily
- avoid accessing the same file server from many clients (farm nodes) in parallel
- Keep tarballs instead of many small files for long-term storage
