This module allows the definition of a custom set of 'ontologies' which can be used to provide autocomplete functionality for a text field, without needing an external server: each ontology is a static list of code/display pairs (and optional synonyms) entered directly into the module's settings. Ontologies can be defined at a site or project level, and a default value can be specified to be returned if no match is found. Since version 0.4, an option has been added to select to swap between word based searching and a full match with all entered text.
Version 0.5 introduces support for synonyms and marking entries as inactive.
For a terser, automatically generated commit-by-commit version record, see CHANGELOG.md.
- PHP 8.0.0 or later
- REDCap 8.8.1 or later, on External Module framework version 16 or later
Dr Daniel Hinostroza from Hospital de Especialidades Carlos Andrade Marín very kindly wrote a Spanish translation for this module, you can find the readme here: Documentación en español Github Documentación en español Thanks to Alejandro Metke Jimenez for updating the Spanish readme for version 0.4 and 0.5; it has continued to be kept in sync with this README since.
The module is licensed under CSIRO Open Source Software Licence Agreement (a variation of the BSD / MIT License).
Default option: download the module from the REDCap External Module Repo
Option 2: download the module from Github and expand the .zip file into the modules folder, matching the version number of the release you downloaded, i.e., redcap/modules/simple_ontology_provider_v1.0.0. The module will then become visible within the REDCap external modules.
In both cases, this should allow configuration and the added benefit of receiving notifications whenever the module is updated within the Repo.
The settings for the module are used for specifying ontologies. If an ontology is added in the site settings for the module it will be available for all projects. Project specific ontologies are added as project settings for the module. If an ontology with the same category is defined in both the site and project settings the project settings will take precedence.
Any number of ontologies can be added, using the follow fields:
Ontology Category- This is the internal name for the ontology, and should be unique for the ontology.Ontology Name- This is the name that will be presented when choosing the ontology in the online designer.Search Type- This drop down is used to select the search algorithm to use. The choices are:Word Based- This is the search mechanism introduced in version 0.3, each word is search separately.Full Match- The full text is searched with no special behaviour for words.
Return all values- Browse a short, fully-enumerated category without needing to guess its exact search wording, instead of requiring the typed text to actually match. See Return all values regardless of search text below for the full explanation, including its interaction withReturn 'No Results Found'.Priority Codes- One code per line; a matching entry whose code is listed here sorts to the top of that category's results, in the order listed. See Priority Codes below for the full explanation.Return 'No Results Found'- This check box is used to indicate that a special value should be returned if no values are returned by a search. The purpose of this is to allow the option to be selected and then have an additional field get activated via branching logic to receive additional data. It can also be used to fall back to some default value.No Results Label- The display value for the special value returned if thereturn no results foundoption is enabled.No Results Code- The value for the special value returned if thereturn no results foundoption is enabled.'No Results Found' result count- An integer value indicating the search result count under which the special no result found value will be included in the results. For example a value of 5, would mean if the search found 5 or fewer matches than the no result value also appear in the search results. The search will only return a maximum of 20 results even if there are more matches, this setting cannot exceed 19.Values Type- This is used to indicate how the values will be provided. The options are:list- A list of values, seperated by a new line. The value and display will be the same value.bar- A list of value|display using a '|' as the separator. Entries are separated with a new line.json- A json array of objects with a code and display. The json can contain other fields, only code and display will be used. Example json:
[
{ "code" : "First Doctor", "display" : "William Hartnell", "period" : "1963-11-23 1966-10-29" },
{ "code" : "Second Doctor", "display" : "Patrick Troughton", "period" : "1966-10-29 1969-06-21" },
{ "code" : "Third Doctor", "display" : "Jon Pertwee", "period" : "1970-01-03 1974-06-08" },
{ "code" : "Forth Doctor", "display" : "Tom Baker", "period" : "1974-06-08 1981-03-21" },
{ "code" : "Fifth Doctor", "display" : "Peter Davison", "period" : "1981-03-21 1984-03-16" },
{ "code" : "Sixth Doctor", "display" : "Colin Baker", "period" : "1984-03-16 1986-12-06" },
{ "code" : "Seventh Doctor", "display" : "Sylvester McCoy", "period" : "1987-09-07 1989-12-06" },
{ "code" : "Eighth Doctor", "display" : "Paul McGann", "period" : "1996-05-27 1996-05-27" },
{ "code" : "Ninth Doctor", "display" : "Christopher Eccleston", "period" : "2005-03-26 2005-06-18" },
{ "code" : "Tenth Doctor", "display" : "David Tennant", "period" : "2005-06-18 2010-01-01" },
{ "code" : "Eleventh Doctor", "display" : "Matt Smith", "period" : "2010-01-01 2013-12-25" },
{ "code" : "Twelfth Doctor", "display" : "Peter Capaldi", "period" : "2013-12-25 2017-12-25" },
{ "code" : "Thirteenth Doctor", "display" : "Jodie Whittaker", "period" : "2017-12-25" }
]
Values- The actual values in the set. The format will depend on the Values Type chosen. Forjson, each entry'scodeanddisplaymust be a plain string or number - not an array or object - and is rejected at save time otherwise.
If the "Select Local Ontology to use" section never appears when editing a text field in the Online Designer - even
though a category is correctly configured - check Control Center's General Configuration page, under
"Can REDCap server access the web (make outbound HTTP calls)?". If this is set to "No, REDCap server does not
have access to the World Wide Web", REDCap core's OntologyManager::buildOntologySelection() returns nothing at
all for every registered ontology provider, before ever calling into any of them - confirmed by reading that
method directly (global $allow_outbound_http; if (!$allow_outbound_http) return '';).
This applies even though this module makes no outbound HTTP calls itself - it is a purely local, static-list provider. REDCap core's own check has no way to distinguish "an ontology provider that needs the internet" (the built-in BioPortal integration) from "one that doesn't" (this module); it disables the entire mechanism site-wide based on a setting that, for administrators who have deliberately locked down outbound access for unrelated reasons, has nothing to do with this module at all. There is nothing this module (or any other ontology provider external module) can do about this from outside REDCap core.
Version 0.5 of the module introduces support for synonyms. A synonym is an alternative text representation for the code. The synonym text will also be search, but the display will always be returned.
Synonyms can be added using the bar and json formats. For the Bar format, the synonyms are added by using more bars to separate the synonyms. For example:
fan|Robert Fanning|Bob Fanning|fan
che|Justine Brown|Justine Chen|Jai Li Chen|che
coo|Cooper Derricks|coo
col|Morty Cole|col
Adds the code as a search term, but also adds alternative versions of names. Any number of synonyms can be added, but for bar delimited they still must be on the same line. For json format, the attribute 'synonyms' is used to provide an array of synonyms.
[
{"code": "fan", "display": "Robert Fanning", "synonyms": ["Bob Fanning", "fan"]},
{"code": "che", "display": "Justine Brown", "synonyms": ["Justine Chen", "Jai Li Chen", "che"]},
{"code": "coo", "display": "Cooper Derricks", "synonyms": ["coo"]},
{"code": "col", "display": "Morty Cole", "synonyms": ["col"]}
]This is the equivalent in json format.
Version 0.5 of the module introduces support for an active flag being associated with an entry. An entry set to active=false will not appear in the searched list, but is still a member of the ontology and so will appear if the value has been used in the past.
How an entry is marked as not active depends on the format used to enter the values.
If the values type is list or bar then the entry is marked as active=false by adding a ! character at the
start of the line. If the code starts with a ! then is can be escaped with a \
fan|Robert Fanning|Bob Fanning|fan
che|Justine Brown|Justine Chen|Jai Li Chen|che
coo|Cooper Derricks|coo
!col|Morty Cole|col
\!escape|Code with an leading !
In the Bar example above, Morty Cole is considered not active and will not show as choice when selecting from the
ontology. The !escape code needs to be escaped with a leading \
For json values, the value is added using an active attribue.
[
{"code": "fan", "display": "Robert Fanning", "synonyms": ["Bob Fanning", "fan"]},
{"code": "che", "display": "Justine Brown", "synonyms": ["Justine Chen", "Jai Li Chen", "che"]},
{"code": "coo", "display": "Cooper Derricks", "synonyms": ["coo"]},
{"code": "col", "display": "Morty Cole", "active": false, "synonyms": ["col"]}
]This is the equivalent in json format. If active is missing or set to something else then the entry will be considered active.
The autocomplete implemented by the module will do a simple text search of the display text for the entered text. In version 0.3 a word based search mechanism was implemented, but in version 0.4 a new option has been added to chose between the word based search or a simple full text match. For the word based search, rather than looking for an exact string match, the module will search for each word in the autocomplete query separately and return all matches sorted by found word count then found position. This behaviour has the side effect that part matches may not be hilighted in the UI. The REDCap autocomplete UI takes the search term used and hilights the matching part of the text in the dropdown.
So if you search for coronav and get a return list of
- Avian Coronavirus
- Bovine Coronanvirus
- Canine CoronanVirus
The 'coronav' section of each choice will be hilighted.
But if you search for 'tuberculosis bacterial' and get a return list of
- 10044772: Tuberculosis of adrenal glands, confirmed by bacterial culture
- 10044780: Tuberculosis of bladder, confirmed by bacterial culture
- 10044799: Tuberculosis of ear, confirmed by bacterial culture
No hilighting will be shown in the UI, as only a full text match is emphasised.
The module parses the codes and displays into an associative array before returning search results. If multiple entries have the same code, then the last entry will overwrite any existing entries.
Both word-based and full-match search require the typed text to actually appear somewhere in an entry's display text or synonyms - which makes sense for a large list, but makes a short, fully-enumerated one (e.g. a frequency scale with values like "Never", "Rarely", "Weekly") hard to browse, since a user has to already know a value's exact wording to find it at all. Checking Return all values on a category removes that requirement: every active, non-hidden entry is always included in that category's results (up to the field's result limit), with entries that do match the typed text still sorted to the top - only their relative order changes, nothing is added or removed from what a normal search would have shown you first.
Interaction with Return 'No Results Found': that setting triggers whenever the result count is below the
field's result limit, not only when there are none (see below) - for a short, fully-enumerated category, which is
exactly what Return all values is for, this means its fallback value will typically appear alongside every
search's real results, not just when nothing matches. Checking both settings on the same category is unlikely to
do what you want.
Priority Codes (one code per line) sorts a matching entry to the top of that category's search results, ahead of any other match, in the order the codes are listed - useful for surfacing a category's most commonly picked values first even when several entries match the typed text equally well. Only the code is considered, not the display text or synonyms.
Priority-codes only ever re-sorts among entries that already qualify for that search - listing a code here does
not, by itself, make it appear. Without Return all values, an entry still has to actually match the typed text
first; with Return all values set, it appears the same way every other non-matching entry does, just sorted to
the front if it's also listed as a priority code.
Unlike advanced_fhir_ontology_provider's and redcap_fhir_ontology_provider's own priority-codes, there is no
"extra fetch" setting to configure here: those modules only get back a subset of results from an external FHIR
server, so a priority code needs extra headroom to have a chance of being included at all. This module's values
are always fully local and already all available, so priority-codes here is a pure re-sort of the complete set -
nothing to tune.
- BREAKING: a code's leading/trailing whitespace is now trimmed - Before priority-codes, a code's whitespace
was never compared against anything else - it was accepted and used consistently everywhere it mattered (as the
field's stored value, and as the lookup key for its own label), so an incidental leading/trailing space around a
code (easy to introduce pasting from a spreadsheet, e.g.
C1 |Weekly) was harmless.Priority Codesand@HIDECHOICEare the first things that need a code typed into one setting to match a code parsed from an entirely separateValueslist, and that's where an accidental whitespace difference between the two silently prevents the match.Valuescodes for all three formats (list,bar,json) are now trimmed, so that cross-referencing is reliable. If any existing category'sValueshappens to have a code with such incidental whitespace, and a record has already stored that literal untrimmed value, that record will show its raw stored value instead of the configured label after upgrading (the code-to-label lookup is now keyed by the trimmed code). If you rely on codes that deliberately include leading/trailing whitespace, review your categories'Valuesbefore upgrading.
Support for the @HIDECHOICE action tag was added to this module in version 0.5. This action tag is available for
choice fields to indicate a choice should not be shown. This can be achieved at an global level in this module by
using the active flag to mark an code as not active. The @HIDECHOICE action tag however is specified at a field
level. So the value will only be hidden for the field the action tag is specified for. The set of values to hide is
defined using a comma separated list of code for the values which should be hidden. The module considers all
@HIDECHOICE entries found in the annotations property of the field.
@HIDECHOICE='code1,code2'
Fixed: @HIDECHOICE was silently ignored on every real autocomplete search. getHideChoice()'s in-memory fast
path read the field's annotation from $Proj->metadata[$field]['field_annotation'], but REDCap's real in-memory
project metadata stores it under the raw DB column name misc - field_annotation is a key name that only exists
in REDCap::getDataDictionary()'s own returned array. Because the fast path's presence check
(isset($Proj->metadata[$field])) still succeeded, it never fell through to the (correct) getDataDictionary()
branch - it just silently returned no annotation, and therefore no hidden codes, for every real request. This had
been broken since @HIDECHOICE was introduced in 0.5; it only ever appeared to work in this module's own test
suite, whose fakes made the same field_annotation mistake.
Added: @SIMPLE-ONTOLOGY-HIDECHOICE, a second tag name for the same purpose. @HIDECHOICE is also REDCap's
own built-in action tag (for a different purpose, on real choice fields), and a module-provided action tag whose
name collides with a built-in one is silently dropped from REDCap's own "@ Action Tags" popup rather than shown -
so this module's repurposing of @HIDECHOICE could never be documented there. @SIMPLE-ONTOLOGY-HIDECHOICE is a
new, non-colliding tag name recognized for exactly the same purpose, registered in that popup; both names are
supported and can be freely mixed on the same field:
@SIMPLE-ONTOLOGY-HIDECHOICE='code1,code2'
Piping is not supported, and not currently possible, in either tag's argument (e.g.
@HIDECHOICE='[other_field]' to hide a code chosen by another field's answer). REDCap core's own built-in
@HIDECHOICE resolves piping in its argument via Piping::replaceVariablesInLabel($text, $record, $event_id, $instance, ...), which needs to know which record is currently being edited. This module's field-level tags are
read from inside DataEntry/web_service_auto_suggest.php - the same real endpoint every search on this field hits
- and that endpoint's request never carries a record, event, or instance identifier at all; REDCap core's own
front-end JS only ever sends
term,field, andpidto it. There is no record context available to pipe against from here, regardless of how this module parses the tag, so this isn't a missing feature so much as a limitation of the integration point itself - it would only become possible if a future REDCap version started including record context in that request.
REDCap caches the display text for each code the first time a project uses it, and reads from that cache from then on instead of asking this module again. This means that after you edit a category's values, existing records can keep showing the old display text until the cache is corrected - see issue #10.
Two links are provided to fix this, matching where a category is defined:
- Refresh Ontology Cache (a project page, under this project's External Modules configuration) refreshes cached entries for this project's own project-level categories. Anyone who can already configure this module's project settings can use it.
- Simple Ontology: Refresh Cache (a Control Center page, REDCap admins only) refreshes cached entries for a site-wide category, across every project that has ever cached a value for it. A project that has defined its own project-level category with the same name is skipped here - use that project's own page instead, so its cache is corrected with its own values rather than the site-wide ones.
Both pages work the same way: pick a category, preview the entries whose cached label no longer matches what the category currently defines (a code that no longer exists in the category at all is left alone, since there is nothing correct to replace it with), then apply the correction to the ones you confirm.
If you save a category with different values than it had before, the module remembers this and shows a reminder - both the next time you open this module's configuration dialog, and by pre-selecting that category the next time you open the relevant refresh page - so you don't have to remember to check on your own.
A REDCap admin can turn this whole mechanism off with the system setting Disable ontology cache refresh. Checking it removes both links, stops the save-time reminder, and rejects the underlying actions outright. It's unchecked by default, so the feature is available unless an admin specifically opts out.
As part of release 8.8.1 of REDCap an extension point was added to allow external modules to become an 'Ontology Provider'. These act like the existing BioPortal ontology mechanism, but allow alternative sources. The main function of an ontology provider is to take a search term and return some match of code + display. This module is a very simple example of an external module which provides this functionality.
To become an ontology provider an external module needs to :
- Implement the OntologyProvider interface
- Register with the Ontology Manager
- Register for an every page hook. (The online designer doesn't have its own hooks)
An ontology provider allows a thrid party service such as bioportal or fhir to provide one or more ontologies.
This ontology is specified in the definition of a field as the enum_element and will be persisted using the form service:category
The service name is used to determine which ontology provider to use.
Selecting an ontology for a field is a two step process, firstly a service is selected from the list of available providers and this will allow the selection of a catergory from the service. The provider needs to produce a string which will live in a hidden div for the service that will be shown when the service is shown, this div will contain ui elements for selecting the category from the service. Once a selection if made, the ui needs to call a javascript function update_ontology_selection($service, $category), this will set a hidden form element used to set the value on the field. Additionally the provider may want to include a javascript function which will be called when the field is populated so that the ui can reflect the current selection. This function should take the form _ontology_changed(service, catgeory).
The provider needs to supply a mechanism which will be used by the autocomplete to search the ontology. This method would make any required ajax calls to return a set of values and a label to go with the selection.
interface OntologyProvider {
/**
* return the name of the ontology service as it will be display on the service selection
* drop down.
*/
public function getProviderName();
/**
return the prefex used to denote ontologies provided by this provider.
*/
public function getServicePrefix();
/**
* Return a string which will be placed in the online designer for
* selecting an ontology for the service.
* When an ontology is selected it should make a javascript call to
* update_ontology_selection($service, $category)
*
* The provider may include a javascript function
* <service>_ontology_changed(service, category)
* which will be called when the ontology selection is changed. This function
* would update any UI elements is the service matches or clear the UI elemements
* if they do not.
*/
public function getOnlineDesignerSection();
/**
* Search API with a search term for a given ontology
* Returns array of results with Notation as key and PrefLabel as value.
*/
public function searchOntology($category, $search_term, $result_limit);
/**
* Takes the value and gives back the label for the value.
*/
public function getLabelForValue($category, $value);
Once an external module has an implementation of OntologyProvider the next step is to ensure the provider is registered for use. This is done by using the code
// register with OntologyManager
$manager = \OntologyManager::getOntologyManager();
$manager->addProvider($provider);
In order for the ontology provider to be available the external module needs to register for the redcap_every_page hook. In this module the ontology provider is registered when the module is constructed, so nothing needs to happen during the hook function,as the registration has already taken place.
In the config.json file of the external module add:
"permissions": [
"redcap_every_page_before_render"
],
In the external module add:
public function redcap_every_page_before_render ($project_id ){
}