WebAPI v2.8.2

Fixes

#1739: Vulnerabilities in dom4j
#1814: SqlRender version has been updated to 1.7.0
#1817: Permission checking for priority daimon has been added
#1833: Update geospatial dependency to release 1.0.1
#1836: Filter jobParameters in notifications
#1839: Check endpoint and prevent any stacktrace output
#1841: Remove quotes from list with source keys
#1853: Broken URL(undefined) after clicking PLP/PLE analysis in notification

WebAPI v2.8.1

Fixes

Raised versions of vulnerable dependencies (#1728, #1727, #1737,#1697, #1726, #1743, #1775)
#1754 : Update CIRCE to 1.9.1
#1730 : ORA-00933: SQL command not properly ended
#1729: PLP/PLE Analysis result file missed analysis executable code
#1720: WebAPI fails to start with enabled webapi-hive profile
#1733: WebAPI 2.8.0 build error - .git directory is not found! Please specify a valid [dotGitDirectory] in your pom.xml
#1759: ancestorAndDescendant:get permission is not added to new sources
#1765: Failed to export IR analysis to CSV on Hive DS (UndeclaredThrowableException)
a721214: Fix sql server migration.
#1772: [Geospatial] Map or geospatial tab should not be shown for "non-geo" data sources
#1791: Not enough permissions for evidence

v2.8.0

New Features

Important note: The 2.8 line of WebAPI will be the last release to support Oracle and MS Sql as the WebAPI database platform. (Note CDM databases are not impacted by this). So, for future major/minor releases, you will need to install a Postgres instance. We can look into community driven support for other platforms, but going forward official support will be limited to Postgresql for WebAPI.

Added validation mechanism to different domain entities of WebAPI: IR Analysis, Cohort Characterization, Pathways, etc). These validation messages are presented in the Atlas UI.

Cohort Characterization: new design element 'Distribution Criteria' allowing the creation of distribution statistics from criteria expressions. (#1449)

Data Sources - Data Density: Added Measurement data over time to Data Density Plot. (#1620)

Cohort Pathways: Allow Repeats option allows pathway segments to repeat in the result. By default, this is disabled. (#1692)

Geospatial functionality (#1445)

Use Dockerfile to facilitate development (#1384)

Cohort Sampling: samples can be created for a given cohort definition on a specific source. (#1657) (Thank you Andries Purnal ([email protected]), Florence Giesen ([email protected]), @MaximMoinat and The Hyve

Vocabulary Service: Added endpoitns for cocnept ancestor (#1664)

Server-Side rendering of cohort expressions: Circe expressions are converted into Markdown using Freemarker templates (#1655).

R packages for PLE/PLP were updated to use Hydra (#1643)

Implemented server-side exports of concept sets for Feature Analysis and Cohort Characterization (#1461)

Individual auth providers can be disabled. Additional information can be found in README.md. (#1623)

Several performance improvements on exporting and results retrieval (#1005, #1010)

Notable Fixes

Due to the long duration between 2.7 and 2.8 release, there were too many fixes to list here. A list of fixes can be found here (https://github.com/OHDSI/WebAPI/issues?q=is%3Aissue+is%3Aclosed+milestone%3AV2.8.0)

Migration Notes

Use SqlRender to translate the following SQL for your specific environment and run on all CDM results schemas:

Cohort Caching

Per #1353, users will need to update the results schema to include the new *_cache tables, specifically:

ALTER TABLE @results_schema.cohort_inclusion ADD design_hash int NULL;

ALTER TABLE @results_schema.cc_results ADD aggregate_id int NULL;
ALTER TABLE @results_schema.cc_results ADD aggregate_name varchar(1000);
ALTER TABLE @results_schema.cc_results ADD missing_means_zero int NULL;

IF OBJECT_ID('@results_schema.cohort_cache', 'U') IS NULL
CREATE TABLE @results_schema.cohort_cache (
	design_hash int NOT NULL,
	SUBJECT_ID bigint NOT NULL,
	cohort_start_date date NOT NULL,
	cohort_end_date date NOT NULL
);

IF OBJECT_ID('@results_schema.cohort_censor_stats_cache', 'U') IS NULL
CREATE TABLE @results_schema.cohort_censor_stats_cache (
  design_hash int NOT NULL,
  lost_count BIGINT NOT NULL
);

IF OBJECT_ID('@results_schema.cohort_inclusion_result_cache', 'U') IS NULL
CREATE TABLE @results_schema.cohort_inclusion_result_cache (
  design_hash int NOT NULL,
  mode_id int NOT NULL,
  inclusion_rule_mask bigint NOT NULL,
  person_count bigint NOT NULL
);

IF OBJECT_ID('@results_schema.cohort_inclusion_stats_cache', 'U') IS NULL
CREATE TABLE @results_schema.cohort_inclusion_stats_cache (
  design_hash int NOT NULL,
  rule_sequence int NOT NULL,
  mode_id int NOT NULL,
  person_count bigint NOT NULL,
  gain_count bigint NOT NULL,
  person_total bigint NOT NULL
);

IF OBJECT_ID('@results_schema.cohort_summary_stats_cache', 'U') IS NULL
CREATE TABLE @results_schema.cohort_summary_stats_cache (
  design_hash int NOT NULL,
  mode_id int NOT NULL,
  base_count bigint NOT NULL,
  final_count bigint NOT NULL
);

IF OBJECT_ID('@results_schema.cohort_sample_element', 'U') IS NULL
CREATE TABLE @results_schema.[cohort_sample_element] (
    [cohort_sample_id] int NOT NULL, 
    [rank_value] int NOT NULL, 
    [person_id] bigint NOT NULL, 
    [age] int NULL, 
    [gender_concept_id] int NULL
);

New cache settings in pom.xml:

cache.generation.invalidAfterDays=${cache.generation.invalidAfterDays} 
cache.generation.cleanupInterval=${cache.generation.cleanupInterval}
cache.generation.useAsync=${cache.generation.useAsync} 
(cache.generation.useAsync details: https://github.com/OHDSI/WebAPI/pull/1459)

Cohort Sampling

IF OBJECT_ID('@results_schema.cohort_sample_element', 'U') IS NULL
CREATE TABLE @results_schema.cohort_sample_element(
    cohort_sample_id int NOT NULL,
    rank_value int NOT NULL,
    person_id bigint NOT NULL,
    age int,
    gender_concept_id int
);

Cohort Characterization

The following alters the existing cc_results table, the full definition can be found here (https://github.com/OHDSI/WebAPI/blob/master/src/main/resources/ddl/results/cohort_characterizations.sql)

alter table@results_schema.cc_results ADD aggregate_id int NULL, aggregate_name varchar(1000), missing_means_zero int NULL;

Acknowledgements

Thanks to @pavgra, @ssuvorov-fls, @aklochkova, @AntonStepanof, @chrisknoll, @wivern, @anton-abushkevich, @ekaterinakrivets, @YaroslavTir, @SergeySukhanov, @KSadomtsev, @konstjar, @vlbe, @AnthonyMolinaro, @MaximMoinat (and the team at TheHyve) for their contributions and reviews as part of this release.

v2.7.9

Fixes

• Updated springsource maven URL to use HTTPS.

WebAPI v2.7.8

Fixes

• Fix for users permissions (#1598)
• Impossible to delete Feature Analysis (#1596)
• Concept set opens with an error (#1595)
• PLP/PLE import failed (#1593)
• Characterization import skips features (#1538)
• ACHILLES reports and caching (#1551)
• Application initialization failed for users without "Atlas users " role (#1584)
• Export and import of characterizations are not compatible. (#1497)
• User is not allowed to edit imported concept set which was added after PLE importing (#1546)
• Characterization fails with SQL error in case of wrong Feature Analysis name (#1566)
• WebAPI failed to start in some cases with enabled AD synchronization (#1561)

WebAPI v2.7.7

Fixes

• Growing number of TEMP_COHORT tables in results schema for Netezza DB (#1067)
• Google OAuth component update (#1511)
• WebAPI does not invalidate jobs after restart (#1467)
• Hydra updates for PLP improvements (#1487)
• FATAL ERROR in PLE analysis if the data source name contains a slash (#1542)
• Shiny directory is not created in PLP results (#1541)

WebAPI v2.7.6

Fixes

• Maven dependencies are failing with a 501 error (#1432)
• Cohort pathway fails to execute on PostgreSQL (#1423)
• Updates Hydra to get latest fixes for PLP/PLE skeleton packages (#1425)

WebAPI v2.7.5

Fixes

• Invalid BigQuery SQL from Atlas Cohorts (#1343)
• Heracles analyses fails on BigQuery (#1366)
• Fixes AD groups import fails. User import job params were moved to separate table (#1354)
• Impossible to create a copy of PLP (#1364)

WebAPI v2.7.4

Fixes

• Adds permissions for accessing /cache/clear endpoint (#1303) @pavgra
• Updates circe to latest release 1.8.2 (#1331) @YaroslavTir @pavgra
• Upgrade Impala driver to 2.6 (using JDBC 4.2) (#1338) @pavgra

WebAPI v2.7.3

Fixes

• Cohort Definition
  • Fix compatibility of /cohortdefinition/{id} endpoint (#1229)
  • Fixed saving of Cohort Definitions when criteria requiring narrower CDM range are added (#1263)
• Cohort Characterization
  • Fixes characterization analysis failed on Impala (#1270)
• Cohort Pathways
  • Fixes failed Cohort Pathways on Impala (#1274)
• Incidence Rates
  • Removed produces annotation + minor improvements (#1232)
• Estimation/Prediction
  • Fixed negative controls calculation fails in PLE for Impala (#1283)
  • Fixes masquerading of credentials in PLE / PLP results (#1254)
• Security
  • Removed legacy service which exposed server side request forgery issue (#1236)
  • Added handlers for unexpected errors (#1235)
  • Added client cache directives to response header (#1234)
• System
  • Automatic invalidation for Execution Engine jobs (#1268)
  • Align required Impala driver version to installed version. Make them consistent with README (#1282)
  • Restored Kerberos auth support (#1240)
  • Updated version of sql render (#1238)
  • Fix grouping problem in BigQuery (#1267)
  • Fixed cycling dependencies (#1264)

Migration Notes

Use SqlRender to translate the following SQL for your specific environment and run on all CDM results schemas:

ALTER TABLE @results_schema.pathway_analysis_stats
  ALTER COLUMN target_cohort_count BIGINT NOT NULL;
ALTER TABLE @results_schema.pathway_analysis_stats
  ALTER COLUMN pathways_count BIGINT NOT NULL;