Choosing Appropriate Data Storage

Many factors go into choosing appropriate data storage for your data. The best place to store your research data depends on what you will do with it and what data protections or security frameworks govern it.

The Feinberg School of Medicine (FSM) data storage policy uses different language to describe research data categories and makes specific recommendations about where Feinberg researchers should store their data. Contact fsmhelp@northwestern.edu with questions about this policy and data storage in general.

Consider the following questions when choosing a location to store your research data.

What Policies and Regulations Apply to My Data?

Not all data storage services meet the minimum requirements for all research data. All Northwestern research data is subject to Northwestern University’s Research Data policy and Data Classification policy, which describes categories of data (levels 1 to 4) referenced in the table below. Other policies and regulations may apply.

Northwestern University Data Classification Policy Categories

* Refers only to the technical controls required to store this data type. Level 3 data and above requires other policies and procedures to be fully compliant with contractual or legal requirements they are subject to.

** Cloud services can usually be configured to satisfy most data storage requirements

Note: Using personal accounts on unsupported storage services like Dropbox is not allowed for storing Northwestern research data. Google Drive, including those provided by Northwestern (e.g., u.northwestern.edu) is also not permitted for Level 2 and above data. Please contact researchdata@northwestern.edu with questions and for advice about alternatives.

How Much Storage Do You Need? How Much Will It Cost?

Another factor to consider when evaluating storage options is the amount and type of data files that will be generated or used as part of your project. Tips for considering capacity when choosing data storage, include:

• Keep your estimate simple. Estimating the amount of data you will have during the course of your project can be intimidating. Instead of trying to predict how much storage you will need down to the megabyte, think about the largest files you will produce, roughly how many you will produce, and round up.
• Use multiple storage services to their strengths. For example, SharePoint is excellent for collaborating with your research group, but it cannot accommodate files larger than 250 GB. Store collaborative documents like manuscripts and notes in SharePoint, and find another platform to store your files larger than 250 GB.
• The location of your data will change through the course of your project. For example, your data could be produced in a core facility and stored on RDSS, then moved to Quest for analysis. Results can be integrated into a manuscript in SharePoint.
• Archival storage is often less expensive. Think about putting infrequently accessed data in archival storage, such as Amazon S3 Glacier Deep Archive.

Any associated costs to storing your data can be included in your grant budgets as data management costs.

Cost and Capacity of Northwestern Storage Services

* Microsoft is currently reviewing its policy on charging for storage.  We are tracking any changes here that may impact the community.

** Feinberg School of Medicine Only. Contact fsmhelp@northwestern.edu with questions about your quota.

Is the Data Backed Up?

Backing up your data prevents data loss due to hardware failure, disasters, file corruption, and human error. While many data storage services do this automatically, understanding what you are protected against when using a storage service is important.

Different types of “backup” strategies protect your data from different risks.

Data Protection Features on Northwestern University Storage Service

Replication involves creating a completely separate copy of your data on another server, typically in a distinct geographic location. Replication is critical for disaster recovery or bringing back an entire system from scratch if it goes down due to natural disaster or cyberattack. RDSS combines versioning with replication by copying its snapshots between the Evanston campus and the Chicago campus.

Who Needs Access to Your Data?

Different storage services have different rules about what is possible with regard to sharing your data with collaborators or the public.

Each system has different default permissions for new files and folders.

• Locations that only you have access to by default are great for storing files that no one else should see. Keep in mind that files that are associated with an individual user account will disappear if they leave Northwestern.
• Locations that grant access to a specific group of people by default are great for collaboration and are not tied to a single user account.

These services may have a method to share outside of this default access group. The following table also outlines who has access by default on each service and how to share outside the default access group.

* Sharing here is defined as granting access to people outside the users who have access by default.

** Feinberg School of Medicine Only

What Compute Source Will I Use?

The best place to store your data depends on whether you’re using your computer, a virtual machine (VM), Quest HPC, or the public cloud as a compute source. Some compute sources offer built-in storage or allow you to mount external drives. Others will require you to transfer your data to the source.

See the table below for details on the following storage options for compute resources. If your compute source is not listed, or you have questions, please request a consultation to discuss data storage options. 

* Make sure to keep a copy elsewhere to protect against data loss.
** Select only files and folders you need to prevent sync problems.
*** You must be on campus or on Northwestern’s VPN to access RDSS/FSMResFiles.
**** Use a Northwestern account to use public cloud services. 
***** Increasing the drive size can be costly.

Making Your Decisions

Choosing where to store your data so that it is safe, compliant, and easy to work is harder than it seems. If you need help deciding where to store your data, email researchdata@northwestern.edu, and our data management consultants will set a time to talk about your workflow and discuss options.