WebAPI v2.7.2

Helpful Resources

• For installation and configuration, please refer to the WebAPI Installation Guide.
• For an overview of ATLAS please refer to the Atlas Video Tutorials.

Important Upgrade Notes

• You must configure one of your vocabulary daimon's to have a priority of 1 or higher as part of this release.
• If you have entities (i.e. cohort definitions, concept sets, etc) with names that are 250 or greater characters in length, you may experience an issue whereby the migration script hangs. This is a known limitation described in #1105 (comment). We'd suggest you use SqlRender to translate the query below and run it against your WebAPI database (specifying the ohdsiSchema for your environment) to see if there are any entities with longer names that should be corrected before upgrading:

{DEFAULT @max_length = 250}
select 'concept_set' table_name , concept_set_name entity_name from @ohdsiSchema.concept_set group by concept_set_name having COUNT(*) > 1 AND LEN(concept_set_name) >= @max_length
union
select 'cohort_definition' table_name , name entity_name from @ohdsiSchema.cohort_definition group by name having COUNT(*) > 1 AND LEN(name) >= @max_length
union
select 'cohort_characterization' table_name , name entity_name from @ohdsiSchema.cohort_characterization group by name having COUNT(*) > 1 AND LEN(name) >= @max_length
union
select 'fe_analysis' table_name , name entity_name from @ohdsiSchema.fe_analysis group by name having COUNT(*) > 1 AND LEN(name) >= @max_length
union
select 'pathway_analysis' table_name , name entity_name from @ohdsiSchema.pathway_analysis group by name having COUNT(*) > 1 AND LEN(name) >= @max_length
union
select 'ir_analysis' table_name , name entity_name from @ohdsiSchema.ir_analysis group by name having COUNT(*) > 1 AND LEN(name) >= @max_length
union
select 'estimation' table_name , name entity_name from @ohdsiSchema.estimation group by name having COUNT(*) > 1 AND LEN(name) >= @max_length
union
select 'prediction' table_name , name entity_name from @ohdsiSchema.prediction group by name having COUNT(*) > 1 AND LEN(name) >= @max_length
;

Fixes

• Data Sources
  • Cache data source reports (#1070)
  • Fix concept record count retrieval (#1176)
• Cohort Definition
  • Add more descriptive error if cohort fails to generate (#972)
  • Remove generating jobs for deleted cohort definitions (#916)
  • Split reporting queries into separate files (#385)
  • Fixes retrieval of Heracles treemap drill-down reports (#1126)
  • Fixed error in procedures by type report SQL (#1151)
  • Fixes condition & drug era treemap drilldowns in cohort reporting (#933)
  • Revise concept path in cohort results queries (#1195)
• Cohort Characterization
  • Moved characterization code into external skeleton package (#975)
  • Fixed approach for updating feature analysis (#785)
  • Fixed generation problems on BigQuery (#1120)
  • Replace design hash value from results with human-readable display (#1110)
  • Save analysis information before completion (#1174)
  • Fix generation issues (#1188)
• Cohort Pathways
  • Performance optimizations for retrieving pathways results (#1008)
  • Fixed import (#1056)
  • Fixes permissions for viewing pathway reports (#1812)
  • Proper handling of results with no records (#1190)
• Incidence Rates
  • Display an error if you do not have permissions to create an Incidence Rate analysis (#948)
  • Fixes permission checks when generating IR on a data source (#1130, #1123, #1740)
  • Use full cohort definitions for import/export (#1171)
  • Fixes permissions for imported IR analysis (#1183)
  • Fixes export to csv (#1879)
• Estimation/Prediction
  • Execution Engine integration fixes (#988, #1003,#1009, #1011,#1007,#1023,#1027, #1079,#1093)
  • Work-around for PLE packages for OHDSI/ParallelLogger#2 (#1180)
  • Revised Hydra generated study package for PLE to fix Shiny display issues (#1197)
• Security
  • Fixed role permissions (#989)
• System
  • Ensure all entity names are unique (#1307, #1338, #1146, #1192)
  • Optimize generation retrieval for characterization & pathways (#828)
  • Companion updates for the wiki (#999)
  • Removed non-OHDSI references from pom.xml (#1051)
  • Fixed Oracle DDL results schema (#1037)
  • Add a caching mechanism to the stack (#1070)
  • Utilize escape sequences with LIKE searches in Hibernate (#1153)
  • Fixes for Oracle migrations (#1159)
  • Improved debugging capabilities for Hikari pool of custom auth DB (#1185)
  • Changes log level for execution engine (#1200)

Here is the full list of fixes

Migration Notes

Use SqlRender to translate the following SQL for your specific environment and run on all CDM results schemas:

IF OBJECT_ID('@results_schema.pathway_analysis_codes', 'U') IS NULL
CREATE TABLE @results_schema.pathway_analysis_codes
(
  pathway_analysis_generation_id BIGINT NOT NULL,
  code BIGINT NOT NULL,
  name VARCHAR(2000) NOT NULL,
  is_combo int NOT NULL
);

IF OBJECT_ID('@results_schema.pathway_analysis_paths', 'U') IS NULL
CREATE TABLE @results_schema.pathway_analysis_paths
(
  pathway_analysis_generation_id BIGINT NOT NULL,
  target_cohort_id INTEGER NOT NULL,
  step_1 BIGINT,
  step_2 BIGINT,
  step_3 BIGINT,
  step_4 BIGINT,
  step_5 BIGINT,
  step_6 BIGINT,
  step_7 BIGINT,
  step_8 BIGINT,
  step_9 BIGINT,
  step_10 BIGINT,
  count_value BIGINT NOT NULL
);

Acknowledgements

Thanks to @pavgra, @ssuvorov-fls, @aklochkova, @AntonStepanof, @chrisknoll, @wivern, @fdefalco, @AnthonyMolinaro for their contributions and reviews as part of this release.

WebAPI v2.7.1

Improvements

• Customizable PLE / PLP skeletons @pavgra (#1021)
• BQ support for PLE / PLP @wivern (#1012)
• Set Priority of first daimon to "1" @ssuvorov-fls (#998)

Fixes

• Fixed issue with import of new estimation/prediction @wivern (#1035)
• Fixed results persistence for PLE R script @pavgra (#1026)
• Split EE result files storage into 2 tables and avoid contents retrieval on each request @pavgra (#1024)
• Fixed creation of new Datasource with keyfile @wivern (#1017)
• Fixed PLE & PLP jobs invalidation @pavgra (#1006)
• Fixed permissions for IR report @aklochkova (#991)

WebAPI Version 2.6.0

New Features

• Cohort Characterization (#495, #588) - thanks @pavgra, @chrisknoll, @wivern, @gowthamrao, @davekern
• Cohort Pathways (#522, #608) - thanks @chrisknoll, @wivern, @pavgra, @anthonysena, @pbr6cornell
• New editors for Population Level Effect Estimation & Patient Level Prediction to support large scale study design (#568, #620, #643) - thanks @anthonysena, @schuemie, @jreps, @PRijnbeek, @ericaVoss, @pbr6cornell, @pavgra, @chrisknoll
• Support for CAS authenication (#589) - thanks @chen-regen, @pavgra

Fixes

• Normalize user references in the database to use IDs (#582) - thanks @wivern, @pavgra
• Renamed migration script names to follow conventions (#586) - thanks @chrisknoll, @pavgra
• Fixed hibernate proxy initialization error in coverters (#596) - thanks @wivern, @pavgra
• Fixed AD User Import (#602, #614) - thanks @mpozhidaeva, @pavgra
• Added logging of security-related events (#591) - thanks @aklochkova, @pavgra
• Fixed application.properties (#635) - thanks @aklochkova, @pavgra
• Get cohort definition list leads to N+1 read problem (#625) - thanks @chrisknoll

... and many others which are here: https://github.com/OHDSI/WebAPI/milestone/17?closed=1

Migration Notes

Use SqlRender to translate the following SQL for your specific environment and run on all CDM results schemas:

CREATE TABLE @results_schema.cc_results
(
  type VARCHAR(255) NOT NULL,
  fa_type VARCHAR(255) NOT NULL,
  cc_generation_id BIGINT NOT NULL,
  analysis_id INTEGER,
  analysis_name VARCHAR(1000),
  covariate_id BIGINT,
  covariate_name VARCHAR(1000),
  time_window VARCHAR(255),
  concept_id INTEGER,
  count_value BIGINT,
  avg_value DOUBLE PRECISION,
  stdev_value DOUBLE PRECISION,
  min_value DOUBLE PRECISION,
  p10_value DOUBLE PRECISION,
  p25_value DOUBLE PRECISION,
  median_value DOUBLE PRECISION,
  p75_value DOUBLE PRECISION,
  p90_value DOUBLE PRECISION,
  max_value DOUBLE PRECISION,
  cohort_definition_id BIGINT
);

IF OBJECT_ID('@results_schema.pathway_analysis_events', 'U') IS NULL
CREATE TABLE @results_schema.pathway_analysis_events
(
  pathway_analysis_generation_id BIGINT NOT NULL,
  target_cohort_id INTEGER NOT NULL,
  combo_id BIGINT NOT NULL,
  subject_id BIGINT NOT NULL,
  ordinal INTEGER,
  cohort_start_date DATETIME NOT NULL,
  cohort_end_date DATETIME NOT NULL
);


CREATE TABLE @results_schema.pathway_analysis_stats
(
  pathway_analysis_generation_id BIGINT NOT NULL,
  target_cohort_id INTEGER NOT NULL,
  target_cohort_count INTEGER NOT NULL,
  pathways_count INTEGER NOT NULL
);

CREATE INDEX idx_pathway_analysis_events_combo_id ON @results_schema.pathway_analysis_events (combo_id);

WebAPI v2.7.0

This release aims to fix bugs from the previous releases and to address functionality gaps from v2.6.0.

New Features

• Improved support for Google BigQuery (#786)
• Added support for GitHub oAuth (#709)
• Added ability to execute Estimation & Prediction Studies via the Arachne Execution Engine (#805)

Fixes

There were 149 issues closed as part of this release. Here are a few highlights:

• Fixes for Incidence Rates (#677, #898)
• Fixes for data sources (#698, #702, #720, #749)
• Fixes cancel of cohort generation (#889)
• Fixes for cohort reporting (#317, #318, #350, #368)
• Fixes for cohort pathways (#645, #690)
• Security fixes (#382, #743)

Here is the full list of fixes

Migration Notes

Use SqlRender to translate the following SQL for your specific environment and run on all CDM results schemas:

CREATE TABLE @results_schema.cohort_censor_stats(
  cohort_definition_id int NOT NULL,
  lost_count BIGINT NOT NULL
);

ALTER TABLE @results_schema.cc_results ADD COLUMN strata_id BIGINT;
ALTER TABLE @results_schema.cc_results ADD COLUMN strata_name VARCHAR(1000);

Record count caching

In previous versions of WebAPI, record counts (sources from the ACHILLES_* tables) were retrieved upon startup and stored in memory for all sources configured with a results daimon. With the 2.7 release, only those results daimons with a priority > 0 will have their record counts cached. (#892)

Acknowledgements

Thanks to @pavgra, @chrisknoll, @wivern, @aklochkova, @fdefalco, @ssuvorov-fls, @mpozhidaeva, @NEONKID, @alondhe, @AlexTawse, @vantonov1, @t-abdul-basser for your contributions to this release.

WebAPI Version 2.5.2

Fixes

• Fixes AD users import login case (#605) - thanks @wivern, @pavgra
• AD import issues (#615) - thanks @mpozhidaeva, @pavgra
• Fixed vocab search by concept_code (#624) - thanks @mpozhidaeva, @pavgra
• Added soft delete for data sources into 2.5.2 (#654) - thanks @pavgra
• Fixed equals for SourceDaimon (#664) - thanks @pavgra
• Fixed Condition and Measurement drilldown reports (show no data) (#755) - thanks @wivern
• Fixed Condition Drilldown report failing on RedShift (#765) - thanks @wivern
• Removes DEFAULT column value from results schema DDL (#771) - thanks @wivern
• Improvements for Impala temp tables performance @wivern

WebAPI Version 2.5.0

Features

Added 'test connection' function for datasource management, exposed in Atlas UI.
Added role management features to import users and roles from AD.
Lazy Loading of Included Concept Set Counts
Added CEM 1.0 services for negative control generation.

Fixes

Better Impala support
Handle issue with null columns in SOURCE table related to username/password storage.
Enhanced AD/LDAP Support

Migration

We've upgraded the Flyway migration library to version 4. It is recommended you backup your WebAPI database prior to deployment. You should see output in your startup logs about a schema migration. Please report any issues you find during upgrade.

Use SqlRender to translate the following SQL for your specific environment and run on all CDM results schemas:

ALTER TABLE @resultsSchema.cohort_inclusion_result ADD mode_id  int NOT NULL DEFAULT 0;
ALTER TABLE @resultsSchema.cohort_inclusion_stats ADD mode_id  int NOT NULL DEFAULT 0;
ALTER TABLE @resultsSchema.cohort_summary_stats ADD mode_id  int NOT NULL DEFAULT 0;

WebAPI Version 2.4.4

Enhancements

• Added ability to filter cost util reports by periods (#532) @agackovka
• Utilization - Optional roll-ups for concept_ancestor (#544, OHDSI/Atlas#836) @gowthamrao, @pavgra

Fixes

• Fixed heracles heel 'monthly change > 100%' analysis for Netezza (84dfb26) @pavgra
• Switched date formatting to classic format without comma (change reason: to properly work in FF) (eb268ce) @pavgra
• Fixes for Achilles and Heracles reports in Impala (#539) @pavgra
• Calculate only summary for Cost & Util when periods are not passed (9afa2f4) @pavgra
• Fixed retrieval of generated cohort members (62a8fc0) @pavgra
• Fixed related concepts retrieval in Impala (#543) @pavgra
• Removed dummy "No matching concept" option in filters for Cost & Util reports (#556) @pavgra

WebAPI Release 2.4.3

Features

• Added Google IAP auth provider (#479) @pavgra

Fixes

• Fixes for Heracles execution and results retrieval in Impala (#515) @pavgra
• Fix for Heracles execution in Redshift (#516) @pavgra
• Fixes for Cost & Util logic (#480, #486) @gowthamrao @pavgra
• Fix for PLE execution via EE (#487) @pavgra
• Security fixes for job executions ($498) @mpozhidaeva
• Fixed PLE / PLP result display (#484) @agackovka

Migration

If you use WebAPI in secured mode, <security.enabled>true</ecurity.enabled> should be replaced with <security.provider>AtlasRegularSecurity</security.provider>

WebAPI Release 2.4.2

Fixes

• Updated to feature Extraction 2.1.4
• Updated to Circe v1.5.0
• Fixes for Heracles Execution.
• Fixes for Execution engine.

WebAPI Release v2.4.1

Features

• PLE & PLP execution via Arachne Exec Engine (#351) @wivern @agackovka
• Support of Impala data sources (#405, #411, #413, #418, #431) @wivern @pavgra
• Separation of JDBC connection link and username & passwords into separate fields for Data sources (#390, #417) @pavgra @chrisknoll
• Storage of data sources' username & passwords in encrypted way (#390) @wivern
• Invalidation of not finished Cohorts / IRs on startup (#429) @wivern
• Added button to invalidate cohort generation job (#430) @wivern

Fixes

• Fixed concurrent auth token update (#437, #436) @pavgra
• Fixed data querying for Drug at baseline and Drug during cohort period reports (#439) @pavgra
• Fixed Achilles heel execution at Netezza (#446) @pavgra
• PLE name change fails (#412) @pavgra
• Fixed possibility of duplicates presence in sec_role table (#420) @pavgra
• Fixed Hikari timeout with turned on auth (#426) @pavgra