Add a tutorial explaining data managers by wm75 · Pull Request #6642 · galaxyproject/training-material

wm75 · 2026-02-11T12:50:40Z

shiltemann · 2026-02-11T16:10:13Z

oh this is great to have, thanks!

bernt-matthias

Excellent. Just a few comments.

bernt-matthias · 2026-02-11T13:15:13Z

+key_points:
+- Data managers are tools to be run by admins of a Galaxy instance.
+- They automate reference data collection and preparation and they write data table (.loc file) records.
+- In addition to a regular tool wrapper xml file, data managers require several xml config files that define the interaction of the data manager with the Galaxy framework and with the data tables they are supposed to populate.


Strictly only one xml file more compared to tools that use the data table.

bernt-matthias · 2026-02-12T08:38:00Z

+# What are data managers and why are they needed?
+
+Many tools run with two kinds of input data: some experimental data specific to
+the tool run (like, e.g., sequencing data) and another dataset, which stays the


Suggested change

the tool run (like, e.g., sequencing data) and another dataset, which stays the

the tool run (like, e.g., sequencing data) and other data, which stays the

bernt-matthias · 2026-02-12T08:41:10Z

+
+One possible solution (which was actually used in the early days of Galaxy) is to have
+Galaxy server admins collect and prepare commonly used data, store it on the server and
+record the location in a so-called .loc file.


Suggested change

record the location in a so-called .loc file.

record the location in a so-called .loc file. These files are simple tab separated files storing data like paths and metadata.

bernt-matthias · 2026-02-12T08:43:09Z

+> An in-depth, technical explanation of this matter is provided at
+> <https://docs.galaxyproject.org/en/latest/dev/data_managers.html>
+> and, when in doubt, that material should be considered the reference document
+> for data managers.


Maybe we can also xref the admin docs: https://docs.galaxyproject.org/en/latest/admin/data_tables.html

That page is already referenced multiple times from the provided page.

bernt-matthias · 2026-02-12T08:45:04Z

+With more and more tools requiring data of very different formats, this approach becomes increasingly
+unmanageable because, for each tool, an admin has to research where to obtain the data,
+or how to calculate it, and whether it needs some reformatting or other pre-processing before being usable
+by tools.


Another advantage is more consistency between different Galaxy servers.

The reproducibilty aspect is now mentioned at the end of the section.

bernt-matthias · 2026-02-12T11:42:07Z

+
+# How does a Data Manager communicate with Galaxy?
+
+1. It declares itself a Data Manager:


Suggested change

1. It declares itself a Data Manager:

1. It declares itself a Data Manager:

which is essentially a normal tool, but uses `tool_type="manage_data"`

bernt-matthias · 2026-02-12T11:44:46Z

+    </data_managers>
+    ```
+
+   This file declares column names for a single Data table (`cat_database`) that


Would it be better to stick to one example? ... But the example is also a good one...

Left as is for now - though I agree that there is maybe room for improvement here.
Sth to address in an update perhaps.

bernt-matthias · 2026-02-12T11:55:03Z

+   Galaxy to record the `extra_files_path` folder name both in the value column and in the
+   name column of the `cat_database` Data table.
+
+   It also wants to store the paths to the extracted `database_folder` and `taxonomy_folder` so that tools that later want to use that data can discover


so that -> such that? (at multiple places?)

bernt-matthias · 2026-02-12T11:57:25Z

+
+   The definitions of the `database_folder` column hold two types of instructions for Galaxy:
+
+   1. The `<move>` element says that Galaxy should take (see the `<src>` element) the data that lives where the `${database_folder}` item of the `data_manager_json` output says it lives and move it to a destination `CAT/${database_folder}` under the base path indicated by `${GALAXY_DATA_MANAGER_DATA_PATH}` (which itself is the configured cached data storage path of the Galaxy instance).


Suggested change

1. The `<move>` element says that Galaxy should take (see the `<src>` element) the data that lives where the `${database_folder}` item of the `data_manager_json` output says it lives and move it to a destination `CAT/${database_folder}` under the base path indicated by `${GALAXY_DATA_MANAGER_DATA_PATH}` (which itself is the configured cached data storage path of the Galaxy instance).

1. The `<move>` element says that Galaxy should take (see the `<src>` element) the data that lives where the `${database_folder}` item of the `data_manager_json` output says it lives and move it to a destination `CAT/${database_folder}` under the base path indicated by `${GALAXY_DATA_MANAGER_DATA_PATH}` (which itself is the admin configured data storage path of the Galaxy instance).

bernt-matthias · 2026-02-12T12:00:04Z

+> If you're following the standard layout of data managers with the tool xml
+> file in a subfolder, you need to run planemo from the parent folder to have
+> it discover all required files beyond the tool xml, but point it to the tool
+> xml to test or serve like, e.g.: `planemo serve data_manager/my_dm.xml`


I always run from the IUC repo root folder. Which is then maybe a bit easier for the serve of DM + tool.

provided now as an example, too

paulzierep

need to continue later

paulzierep · 2026-02-12T13:57:38Z

+---
+layout: tutorial_hands_on
+
+title: Understanding Galaxy Data managers


Suggested change

title: Understanding Galaxy Data managers

title: Understanding Galaxy Data Managers

paulzierep · 2026-02-12T14:02:00Z

+because:
+
+- some of that data is complicated to gather from public sources or needs some pre-processing
+- it leads to unnecessary copies of data that would better be reused across user accounts


Suggested change

- it leads to unnecessary copies of data that would better be reused across user accounts

- it leads to unnecessary copies of data (that can easily be more then 100 GB of size) that would better be reused across user accounts

paulzierep · 2026-02-12T14:06:24Z

+
+![The issue of maintaining different kinds of data with the .loc files approach illustrated through three examples](../../images/loc-files-approach.png "The issue of maintaining different kinds of data with the .loc files approach")
+
+In principle, admins could automate some of this work through scripts, but it would be nice to not have each admin reinvent the wheel.


Suggested change

In principle, admins could automate some of this work through scripts, but it would be nice to not have each admin reinvent the wheel.

In principle, admins could automate some of this work through scripts, but it would be nice to not have each admin reinvent the wheel. Furthermore, since it is in the intererst of researchers to have the reference data for their tool available, they should be able so support admins in the data collection task trough an open process.

Adds too much advanced perspective, imo. Lets keep things simple here

bernt-matthias · 2026-05-04T11:59:39Z

+
+   1. The `<move>` element says that Galaxy should take (see the `<source>` element) the data that lives where the `${database_folder}` item of the `data_manager_json` output says it lives and move it to a destination `CAT/${database_folder}` under the base path indicated by `${GALAXY_DATA_MANAGER_DATA_PATH}` (which itself is the configured cached data storage path of the Galaxy instance).
+
+   2. The first `<value_translation>` element says that Galaxy should not write the Data manager tool-provided value for `database_folder` directly, but instead first translate it to `${GALAXY_DATA_MANAGER_DATA_PATH}/CAT/${database_folder`. If you compare the resulting string with the `<move>` instructions, you will see that it will now be the same as the ultimate path to the folder after Galaxy has moved it.


Still struggling to understand value_translation. Why is here https://github.com/natefoo/tools-iuc/blob/bd4a77513afed69f727038cdd11647daddd41c2f/data_managers/data_manager_bwa_mem_index_builder/data_manager_conf.xml#L14 the copy target different from the value translation?

Also there is a missing closing parenthesis in ${GALAXY_DATA_MANAGER_DATA_PATH}/CAT/${database_folder

So, in the case of bwa mem, the DM creates the index dir at ${out_file.extra_files_path}/${fasta_file_name}, see:

https://github.com/natefoo/tools-iuc/blob/bd4a77513afed69f727038cdd11647daddd41c2f/data_managers/data_manager_bwa_mem_index_builder/data_manager/bwa_mem_index_builder.xml#L18

and records ${fasta_file_name as the path in:

https://github.com/natefoo/tools-iuc/blob/bd4a77513afed69f727038cdd11647daddd41c2f/data_managers/data_manager_bwa_mem_index_builder/data_manager/bwa_mem_index_builder.xml#L22-L38

the move operation you're linking to then moves everything under ${out_file.extra_files_path} including the index dir to the destination <target base="${GALAXY_DATA_MANAGER_DATA_PATH}">genomes/${dbkey}/bwa_mem_index/v1/${value}</target>, but the data table needs to record the entire path to the index dir (named after ${fasta_file_name) so it appends $path to the value_translation.
There are almost always several ways to achieve the same kind of result so probably one could also rewrite this one in a way that target and value_translation are identical.

Thanks for pointing out the missing parenthesis.

wm75

Perhaps ready for merging before the tool-dev workshop?

wm75 · 2026-09-24T12:19:30Z

+> An in-depth, technical explanation of this matter is provided at
+> <https://docs.galaxyproject.org/en/latest/dev/data_managers.html>
+> and, when in doubt, that material should be considered the reference document
+> for data managers.


That page is already referenced multiple times from the provided page.

wm75 · 2026-09-24T12:20:54Z

+
+![The issue of maintaining different kinds of data with the .loc files approach illustrated through three examples](../../images/loc-files-approach.png "The issue of maintaining different kinds of data with the .loc files approach")
+
+In principle, admins could automate some of this work through scripts, but it would be nice to not have each admin reinvent the wheel.


Adds too much advanced perspective, imo. Lets keep things simple here

wm75 · 2026-09-24T13:25:54Z

+
+5. A `tool-data` folder
+
+   Here, tools can provide samples of the .loc files they are going to work with.


rewrote this section

wm75 · 2026-09-27T12:12:19Z

+With more and more tools requiring data of very different formats, this approach becomes increasingly
+unmanageable because, for each tool, an admin has to research where to obtain the data,
+or how to calculate it, and whether it needs some reformatting or other pre-processing before being usable
+by tools.


The reproducibilty aspect is now mentioned at the end of the section.

wm75 · 2026-09-27T12:18:33Z

+    </data_managers>
+    ```
+
+   This file declares column names for a single Data table (`cat_database`) that


Left as is for now - though I agree that there is maybe room for improvement here.
Sth to address in an update perhaps.

wm75 · 2026-09-27T12:23:46Z

+> If you're following the standard layout of data managers with the tool xml
+> file in a subfolder, you need to run planemo from the parent folder to have
+> it discover all required files beyond the tool xml, but point it to the tool
+> xml to test or serve like, e.g.: `planemo serve data_manager/my_dm.xml`


provided now as an example, too

bernt-matthias · 2026-09-28T15:02:36Z

+   > e.g. by uploading the tool to a Galaxy Toolshed, you must *never* change its layout again in a later version of that tool, or in a new tool reusing the Data Table!
+   > No renaming of columns, no reshuffling, and no addition of new columns! You need to keep that layout frozen!
+   >
+   > It is, therefor, important to consider a new Data Table's columns very carefully.


maybe add thw new sectio in the iuc guidelines maybe xref the new iuc standard somewhere galaxy-iuc/standards#93

both are very good suggestions, thanks!

bernt-matthias · 2026-09-28T15:12:05Z

+               'name': '<extra_files_path>',
+               'taxonomy_folder': '<extra_files_path>/a_taxonomy',
+               'value': '<extra_files_path>'


Should we use extra_files_path name and value in the example?

We should explain the name and value columns

Kind of addressed now by referencing the IUC column name standards above.
Maybe you're right about this not being the perfect example for using name and value.
I'd leave this for future improvements.

Add data managers tutorial

2ab1209

github-actions Bot added the dev label Feb 11, 2026

wm75 added 2 commits February 11, 2026 18:54

Fix time estimation metadata

3a45766

... again :-)

edc3e90

bernt-matthias reviewed Feb 12, 2026

View reviewed changes

paulzierep reviewed Feb 12, 2026

View reviewed changes

Fix <source> elements

c140f19

wm75 mentioned this pull request Mar 19, 2026

New tool addition: Samestr2 galaxyproject/tools-iuc#7304

Merged

3 of 5 tasks

bernt-matthias reviewed May 4, 2026

View reviewed changes

SaimMomin12 and others added 4 commits July 28, 2026 14:10

Merge branch 'main' into add-dm-tutorial

a0968a9

Extend and address first round of review comments

bfe3a78

Harmonize capitalization of Data Manager (Tool) / Data Table

1d06896

Complete the DM testing section of the tutorial

de7e615

wm75 commented Sep 27, 2026

View reviewed changes

shiltemann marked this pull request as ready for review September 28, 2026 09:35

shiltemann requested review from bebatut and lecorguille as code owners September 28, 2026 09:35

bernt-matthias previously approved these changes Sep 28, 2026

View reviewed changes

Reference IUC Data Table Standards for common column names

e221bae

wm75 dismissed bernt-matthias’s stale review via e221bae September 29, 2026 16:25

	the tool run (like, e.g., sequencing data) and another dataset, which stays the
	the tool run (like, e.g., sequencing data) and other data, which stays the

	record the location in a so-called .loc file.
	record the location in a so-called .loc file. These files are simple tab separated files storing data like paths and metadata.


		# How does a Data Manager communicate with Galaxy?

		1. It declares itself a Data Manager:


		The definitions of the `database_folder` column hold two types of instructions for Galaxy:

		1. The `<move>` element says that Galaxy should take (see the `<src>` element) the data that lives where the `${database_folder}` item of the `data_manager_json` output says it lives and move it to a destination `CAT/${database_folder}` under the base path indicated by `${GALAXY_DATA_MANAGER_DATA_PATH}` (which itself is the configured cached data storage path of the Galaxy instance).

	title: Understanding Galaxy Data managers
	title: Understanding Galaxy Data Managers

	- it leads to unnecessary copies of data that would better be reused across user accounts
	- it leads to unnecessary copies of data (that can easily be more then 100 GB of size) that would better be reused across user accounts


		![The issue of maintaining different kinds of data with the .loc files approach illustrated through three examples](../../images/loc-files-approach.png "The issue of maintaining different kinds of data with the .loc files approach")

		In principle, admins could automate some of this work through scripts, but it would be nice to not have each admin reinvent the wheel.


		1. The `<move>` element says that Galaxy should take (see the `<source>` element) the data that lives where the `${database_folder}` item of the `data_manager_json` output says it lives and move it to a destination `CAT/${database_folder}` under the base path indicated by `${GALAXY_DATA_MANAGER_DATA_PATH}` (which itself is the configured cached data storage path of the Galaxy instance).

		2. The first `<value_translation>` element says that Galaxy should not write the Data manager tool-provided value for `database_folder` directly, but instead first translate it to `${GALAXY_DATA_MANAGER_DATA_PATH}/CAT/${database_folder`. If you compare the resulting string with the `<move>` instructions, you will see that it will now be the same as the ultimate path to the folder after Galaxy has moved it.


		5. A `tool-data` folder

		Here, tools can provide samples of the .loc files they are going to work with.

Conversation

wm75 commented Feb 11, 2026

Uh oh!

shiltemann commented Feb 11, 2026

Uh oh!

bernt-matthias left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paulzierep left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wm75 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!