Scientific, research, and laboratory software
Research Data Collaboration Platform Requirements Guide
A practical engineering guide for research organizations replacing scattered drives, ad hoc transfers, undocumented analysis, and fragile publication handoffs with governed data infrastructure.
Published by Kennedy Gichobi · Fact-checked by OpenAI Codex research review · Published · 1849 words
Define the research lifecycle before choosing storage
A research data platform should make scientific work more findable, interpretable, reproducible, protectable, and transferable. It is not merely a large shared drive. Begin with the decisions teams cannot make reliably: which dataset is authoritative, how it was produced, who may use it, whether consent or agreements permit sharing, which analysis generated a figure, what must be preserved, and who owns stewardship after a project ends.
Map representative journeys from proposal and data-management planning through project setup, collection or acquisition, transfer, validation, organization, description, transformation, analysis, collaboration, quality review, publication, repository deposit, controlled access, reuse, correction, retention, preservation, and disposition. Include changed protocol, withdrawn participant, instrument reprocessing, collaborator departure, embargo, external repository outage, software dependency loss, corrected paper, and a dataset too large to copy casually.
Qualified investigators, data stewards, research administrators, ethics and privacy owners, information security, librarians or archivists, statisticians, domain specialists, legal advisers, funders, and institutional officials must define policy. Software should implement approved plans and evidence; it cannot decide that all data should be open or that de-identification removes every privacy risk.
Model research objects and relationships explicitly
Represent program, award, project, study, protocol, person, organization, role, site, instrument, method, sample or subject reference, collection event, dataset, file, table, variable, code, environment, workflow, run, result, figure, publication, plan, consent category, agreement, access decision, repository deposit, identifier, retention rule, and provenance event separately where relevant.
Use stable internal identifiers and support persistent external identifiers without making a filename or paper title the database key. One dataset may support several analyses and publications; one result may combine several datasets and code versions; one researcher may change name or institution. Preserve those many-to-many relationships.
Define dataset states such as active, validation, analysis-ready, frozen, released, superseded, restricted, embargoed, archived, and scheduled for disposition. State meanings, permitted transitions, authority, and version behavior must be explicit. A folder called final cannot distinguish a publication snapshot from the latest corrected working data.
Treat metadata and provenance as core data
Define required descriptive, structural, administrative, technical, rights, preservation, and domain metadata by data type and lifecycle stage. Capture units, vocabularies, coordinate systems, missing-value codes, instruments, methods, software, parameters, quality flags, and known limitations. Validate metadata at creation and transfer rather than waiting until repository deposit.
The NIST Research Data Framework describes metadata and provenance as essential to effective use, reuse, and preservation. Use its lifecycle questions as a planning resource, then select community standards appropriate to the discipline. A universal metadata form often produces technically complete but scientifically meaningless records.
Record provenance as attributed events: source, acquisition, import, validation, transformation, combination, subset, analysis, correction, and release. Link inputs, process or code version, parameters, environment, actor or service, time, outputs, warnings, and quality decision. Preserve the original and transformation history instead of overwriting a derived file in place.
Design data-management plans as living operational records
Represent a plan's scope, data types, formats, metadata, tools, standards, storage, security, access, sharing, repository, timeline, restrictions, preservation, costs, responsible roles, approval, and revision history. Link commitments to executable tasks and evidence. A document submitted with a proposal becomes useful only when teams can see what is due and whether practice changed.
NIH's Data Management and Sharing Policy applies to covered NIH-funded or conducted research. Current 2026 plan elements emphasize appropriate sharing, timing, duration, limitations, established repositories, and special genomic-data considerations. Other funders and institutions have different requirements. Store the relevant authority and effective version rather than applying an NIH template universally.
Support reviewed plan changes during the project. New data types, consent limits, repository constraints, collaborator terms, or scientific methods may require amendment. Preserve who approved the revision, its effect on prior data, costs, and reporting. Do not silently edit a plan after progress reporting.
Make computation reproducible enough for the decision
Connect code repositories, notebooks, workflows, containers or environments, package locks, configuration, secrets references, compute jobs, logs, inputs, outputs, and review. Record the exact commit or release, environment image, parameters, random seeds where relevant, hardware dependency, and execution time needed to interpret a result.
Not every exploratory action needs production-grade orchestration, but evidence supporting a publication, regulatory submission, safety conclusion, or costly follow-up requires proportionate reproducibility. Define the threshold at which an exploratory notebook becomes a controlled analysis and preserve the promotion decision.
Separate source code from data and secrets. Prevent credentials, restricted records, or large outputs from entering general code repositories. Support approved compute near controlled data when copying would increase risk or cost. Capture failures and warnings, not only successful runs.
Govern access from purpose, agreements, and data risk
Authorize by organization, project, role, dataset, purpose, agreement, consent or policy category, sensitivity, stage, and action. Enforce controls on APIs, files, queries, previews, search, exports, compute, reports, jobs, and support tools. Search results and counts must not reveal restricted studies or participant attributes.
Model data-use agreements, material or collaboration agreements, approvals, training, conflicts, embargoes, access requests, decisions, conditions, expiration, renewal, revocation, and attestation. A principal investigator's invitation should not bypass institutional authority. Collaborator departure must revoke access while preserving attributed contributions and required records.
NIH privacy guidance for participant data emphasizes proactive assessment and may support controlled access where appropriate. The exact obligations depend on consent, ethics review, law, policy, community expectations, and agreements. Treat de-identification as a documented risk-control process with residual risk, not a permanent anonymous label.
Support collaboration without erasing scientific accountability
Provide comments, tasks, issues, review requests, annotations, and decisions linked to the specific data, code, result, or version. Make visibility and retention clear. Important rationale should not live only in transient chat, while informal exploration should not automatically become a permanent official record without policy.
Define contributor roles and attribution. Preserve who collected, curated, analyzed, reviewed, approved, shared, and maintained an object. Persistent researcher identifiers can reduce name ambiguity, but identity linking needs user control and correction. Authorship and credit remain scholarly decisions, not automatic output from activity counts.
Handle concurrent edits, branches, merges, locked snapshots, and rejected changes according to data type. A spreadsheet, database, image annotation, and analysis pipeline need different collaboration mechanics. Never present last-write-wins behavior as version control. Provide safe comparison and recovery so a rejected merge or accidental overwrite does not require administrators to reconstruct scientific state directly from storage internals.
Plan repository selection and publication handoff early
Represent repository requirements, accepted formats, metadata, identifiers, license, access mode, embargo, review, size, cost, retention, preservation, API, update, withdrawal, and succession. Test deposit with representative data before the publication deadline. An established disciplinary repository may be preferable to building a public download page.
Generate deposit packages from approved immutable snapshots with manifests, documentation, code or environment references, rights, provenance, and known limitations. Validate the package after repository ingestion. Record the repository identifier, version, landing page, access terms, deposit evidence, and relationship to publications.
Plan correction and withdrawal. A dataset may receive a new version, restricted status, corrected metadata, or a tombstone while citations to the earlier release remain. Preserve the reason, authority, communication, and relationship. Do not replace bytes behind a persistent identifier without the repository's approved version process.
Make discovery useful without exposing restricted information
Index approved titles, descriptions, keywords, investigators, projects, methods, variables, spatial or temporal coverage, identifiers, and relationships. Distinguish public metadata, institution-visible records, project-only records, and undiscoverable sensitive work. Apply access before counts, suggestions, snippets, and related-item recommendations are generated.
Support fielded search, filters, vocabularies, saved queries, and dataset comparison. Show metadata completeness, version, freshness, access route, and known limitations. A relevance ranking should not make popular datasets appear more scientifically appropriate than specialized or recent material.
Measure successful discovery and reuse with privacy-aware metrics, feedback, citation, access fulfillment, and support data. Avoid surveillance of researcher interests or using download counts as a proxy for scientific quality. Document metric definitions and access boundaries, suppress unsafe small-group reporting, and let data stewards distinguish a discovery problem from unavailable access, insufficient documentation, or a legitimately specialized audience.
Integrate instruments, storage, identity, and research administration
Inventory collection tools, instruments, LIMS, electronic notebooks, survey systems, code hosting, compute clusters, cloud storage, identity, grants, ethics review, agreements, repositories, publication systems, and preservation services. For each integration, define authority, identifiers, formats, units, schema version, permission, transfer method, retries, reconciliation, support, retention, and exit.
Do not mirror every system into an uncontrolled data lake. Move only what the research purpose and approved architecture require. Expose failed transfers, schema drift, missing files, expired approvals, quota limits, mismatched investigators, and rejected deposits in accountable queues.
Use service identities with narrow permissions, managed secrets, rotation, and auditable ownership. Researcher personal tokens should not become permanent production infrastructure. Confirm that the institution can export metadata, files, relationships, audit evidence, and access decisions if a vendor or project ends.
Plan cost, scale, preservation, and environmental tradeoffs
Estimate storage by data type, growth, replication, versioning, processing copies, transfer, retrieval, backup, preservation, and repository deposit. Include compute, egress, specialist curation, metadata work, privacy review, software maintenance, and support. Cheap hot storage today can create unaffordable retention later.
Apply lifecycle policies only after scientific, legal, funder, agreement, and preservation owners define what may move or be disposed. Ensure archived data remain discoverable and retrievable within the promised time. Test restoration of a complete research object, including metadata, relationships, code, and permissions.
Measure infrastructure use and eliminate needless duplication where practical without compromising resilience or obligations. Sustainability claims should state boundaries and assumptions; storage class alone does not prove a research platform is environmentally responsible. Include compute efficiency, data movement, idle resources, retention choices, hardware lifecycle, provider evidence, and the scientific value or obligation that justifies continued resource use.
Validate migration and acceptance with a difficult research scenario
Profile shared drives, object stores, workstations, notebooks, code repositories, databases, instrument exports, and legacy catalogs. Measure duplicate and near-duplicate datasets, broken paths, missing metadata, unknown units, orphaned owners, inconsistent identifiers, inaccessible formats, embedded secrets, unclear consent, and files that cannot be tied to a study.
Migrate in governed batches with mappings, rejection, manifests, fixity checks, relationship reconciliation, sample scientific review, rollback, and legacy access. Do not label unexplained folders as authoritative datasets merely to complete a count. Require domain owners to approve representative reconstructed studies, including their metadata, provenance, code relationships, permissions, and publication links, before retiring the source environment.
Ask a vendor or developer to demonstrate a demanding scenario: a collaborator uploads a large dataset twice, metadata conflict with the protocol, analysis uses an undocumented environment, consent limits one subset, the investigator changes institution, a repository rejects the package, a paper is corrected, and preserved data must be restored years later. A strong system explains identity, version, provenance, authority, access, cost, and recovery throughout.
Review the related laboratory information management checklist for sample, method, instrument, and result workflows. Share disciplines, projects, data types, scale, metadata standards, privacy boundaries, compute, repositories, integrations, migration sources, and preservation needs through the project questionnaire, or use quick contact for a focused question.
Authoritative references
Related software planning guides
- Biotechnology Research Workflow Platform Requirements
- Laboratory Information Management System Requirements Checklist
- Laboratory Software Data Migration and Validation Guide