Release Notes

5.3.0 - June 16, 2020

OmniSci release 5.3.0 contains several new features and improvements and fixes a number of issues.

OmniSciDB

New Features and Improvements

• Array support for the Array type over JDBC.
• Added support for SELECT DISTINCT in UNION ALL. UNION ALL is currently prerelease and must be explicitly enabled in the server configuration file.
• Added a query type return to the sql_execute endpoint, allowing API users to determine if a query altered date or a table or was a read-only query.
• Added support for joins on DECIMAL types.
• Added support for retrieving metadata for a given statement after running SQLPrepare in ODBC. For example, after calling SQLPrepare, the SQLColAttribute will return metadata for columns involved in the query.
• Performance improvements on CUDA GPUs, particularly Volta and Turing. OmniSci now generates PTX code targeted at specific GPU architectures, resulting in increased performance for aggregate workloads. Performance of summation on double precision types has increased by 10x in some instances.
• Increased use of CUDA shared memory, resulting in additional performance increases on aggregate queries in the presence of a group by.
• sql_validate now returns column descriptors for each projected column in the order in which columns appear in the select clause. Previously, column descriptors could be in any order.

Fixed Issues

• Resolved an issue where RESTORE TABLE could be exploited to gain shell access on the running Omnisci server.
• Resolved an issue where incorrect GPU memory requirement estimates for JOIN queries could result in a query that would run on GPU being transferred unnecessarily to CPU.
• Reduced memory fragmentation during select queries, particularly with approx_count_distinct aggregates. OmniSci recommends dropping caches or restarting the system after upgrading to OmniSci 5.3.0 to make more large pages available for the server.
• Resolved an issue where a comparison of geometry types could crash. Most comparisons of geometry types are not yet supported.
• Resolved various crashes when sorting, projecting, or otherwise interacting with geospatial columns. These crashes resulted from an unsupported operation being performed on geospatial data, and now result in error messages.
• Fixed a crash on Boolean comparisons where the Boolean column was declared NOT NULL.
• Resolved an issue where DISTINCT * in a subquery was not respected, resulting in an extra rowid being inserted into the subquery result and skewing the overall distribution in the query. Now, rowid is not present in a SELECT * statement unless explicitly projected.
• Resolved an issue where a syntax error in a SQL statement could cause the aggregator node to crash in a distributed setup.
• Resolved an issue where a multicolumn group-by window function could return incorrect results.
• Resolved an issue where the date_trunc function could return incorrect results for dates before the Unix epoch, January 1st 1970.
• Resolved an issue where dictionary encoded string columns using a shared dictionary would incorrectly have an encoding displayed in SHOW CREATE TABLE;` or `\d`. Now, DDL from a table with columns using a shared dictionary can be copied and pasted directly to create a new, identical table.
• Resolved an issue where multiple filter expressions using OR could cause a crash if the same column was used on the left side of the expression with different casts.
• Added logging for OmniSciDB server shutdowns due to exceeding the maximum number of available open files on the system.
• Resolved an issue where `INSERT INTO` commands sent from the ODBC driver would fail.
• Fixed an issue where a series of nested queries grouping on float columns could give incorrect results.
• Fixed an issue where importing geojson files with embedded dates could truncate the date epoch to 0 or NULL.
• Resolved an issue where ST_Contains queries using the overlaps join framework could crash or return incorrect results.
• Resolved an issue where importing from a parquet file with multiple row groups could return incorrect results.
• Resolved an issue in the ODBC driver where incorrect data could be returned for double precision types.
• Resolved an issue where geospatial joins could return incorrect results.
• Resolved inconsistencies in argument support for date/time functions. Now, all date/time functions should support literal arguments with or without quotes (for example, `date_part(day, …` and `date_part(‘day’, …)`
• Resolved a small memory leak in the OmniSciDB SQL parser.

OmniSci Render

Fixed Issues

• Resolved an issue where the server could crash when a user disconnected if the users render session had timed out.
• Resolved a crash where switching between different point renders on the same data could cause a crash if hit testing is disabled.

OmniSci Immerse

New Features and Improvements

• New Combo chart with increased configurability and power:
  • New chart configurations like row/column, stacked/percentage bar, stacked/percentage line and area, line/bar combination.
  • For numerical base dimensions, measures can be configured to display as cumulative and percentage distributions.
  • A new “Top N” experience for charts with a group-by dimension, allowing you to select dimension values dynamically or manually (including a new “Add all” option), as well as the ability to configure sorting logic for Top N.
  • Chart legends can be pinned to the right or superimposed on the chart.
  • The visibility of individual group by series can be toggled on or off from the chart legend
  • Chart orientation can be configured between vertical and horizontal.
  • The min and max of chart axes can be independently locked. Existing bar and combo charts can “duplicate as New Combo chart” at the dashboard level.
• Chart-specific filters across all chart types, which can be toggled off when not in use.
• For chart-specific filters that require an exact match on the filter value, you can use “quick filters” to access those filters directly on the chart.
• A simplified filter panel, with two modes: the existing “Advanced” mode, and a new “Simple mode” focused on simplifying the consumption and interactivity of filters.
• An updated filter sets experience targeted at simplifying options and offering greater visibility across filter sets.
• Added sorting by column value for dashboards on the dashboards list.
• On multi-layer charts, visibility of individual map layers can be set by zoom level.
• Different map charts can be synced together for pan and zoom actions, regardless of data source.

Fixed Issues

• Resolved issue with global filters adding escapes to string values with underscores or percent marks.
• Resolved issue in histogram and combo charts where binning min and max values changed after saving and reopening the chart.
• Resolved issue in the SQL Editor where newly created tables did not appear on the tables list.
• Resolved issue in pointmap where Boolean values could not be selected for the color parameter.
• Resolved issue where invalid config did not cause the web server to fail in a visible way.
• Resolved issue where switching database while viewing a dashboard did not allow the user to open a dashboard in the new database.
• Resolved issue with database switching where user permissions were set incorrectly.
• Resolved issue in the filter panel where time-based filters were displaying local time vs. UTC.
• Resolved issue where cohort creation errored out on dashboards containing geo-joined multilayer charts.
• Fixed minor UX and display issues, specific to the dark and light mode themes.

5.2.2 - May 7, 2020

OmniSci release 5.2.2 includes a new feature and fixes two issues.

OmniSciDB

New Feature

• Added support for geospatial columns in temporary tables.

Fixed Issues

• Resolved an issue in which a render request and a table modify operation (for example, COPY FROM, DROP TABLE, TRUNCATE TABLE), dispatched simultaneously, could cause a lockup.
• Fixed an incorrect result for queries that sort based on the result of an aggregate in which the aggregate type is not NULL. This typically occurred with an explicit non-NULL filter in the WHERE clause on the input to the aggregate.

5.2.1 - April 20, 2020

OmniSci release 5.2.1 fixes an issue.

OmniSciDB

Fixed Issue

• Resolves an issue where OmniSciDB fails to respond correctly to operating system signals in some contexts.

5.2.0 - April 16, 2020

OmniSci release 5.2.0 contains several new features and improvements and fixes a number of issues.

OmniSciDB

New Features and Improvements

• CUDA requirement is updated to version 10.1. OmniSciDB now requires NVIDIA driver v418.39 or later.
  • The UDF compiler now requires Clang++ v9.0.
• Updated embedded Apache Arrow to support v0.16.
  • Added INT8 support to Arrow result converter.
• Multi-fragment hash join iterator added to improve performance on multi-fragment inner joins and geospatial joins.
• Added NULL support for geospatial types, including in ALTER TABLE ADD COLUMN.
• Improved performance for query interrupt (currently disabled by default), allowing you to interrupt a query in omnisql (using Ctrl+C) and JDBC (using the Cancel endpoint).
• Added SQL SHOW commands:
  • SHOW TABLES -- Show all tables the user currently has access to in the logged-in database.
  • SHOW DATABASES -- Show all databases the user can log in to.
  • SHOW CREATE TABLE -- Show the DDL CREATE TABLE command for a given table.
  • SHOW USER SESSIONS -- Show all persisted sessions on the server (superuser only). Expired sessions are still shown; they are removed from session state on login attempt.
• Added support for the ALTER TABLE DROP COLUMN command.
• Updates and deletes are now supported on temporary tables.
• Added additional UDF command-line options for systems using nondefault Clang++ installation locations.
• Added support for quoted table names in DDL and DML commands. Use quotes if the table name contains special characters or matches a reserved word.
• Added support for window functions in distributed mode.
• JDBC driver updates:
  • Supports the cancel method, requesting that the database cancel the current request. The enable-runtime-query-interrupt flag must be set to TRUE in OmniSciDB configuration.
  • Handles JDBC function escape syntax for the fn keyword.
  • Supports the getMetaData endpoint.
  • Implements setSchema as a null operation (OmniSciDB currently does not support schemas) instead of throwing a "method not implemented" error.
• Added support for EXTRACT(DATEEPOCH from m), where m is a TIMESTAMP column.
• omnisql now returns a nonzero exit code if the last executed query failed to complete (threw an exception).
• Improved logging of connection failures when using LDAP, SAML, or other external authentication providers.
• Added support for COPY TO in distributed mode.
• Added support for importing compressed parquet files (gzip).
• Various performance improvements, particularly for join queries, projection queries with order by and/or limit, queries with scalar subqueries, and multicolumn group-by queries.

Fixed Issues

• Resolved a series of locking sequence errors, which could cause deadlocks when multiple users were interacting with the database and rendering engine at the same time.
• Improved the ability of omnisci_server to detect a failure in the Calcite service, resulting in automatic restart of omnisci_server when managed by systemd.
• Resolved an issue where projection queries running in columnar mode could crash the server.
• Resolved an issue where queries could fail or crash the server with “error code 8”. Those queries are now gracefully retried or aborted, and a correct error message is now displayed (for example, "not enough GPU memory to execute the query".)
• The geospatial importer is now more tolerant of unsupported types. Instead of causing the import to abort, unsupported types in a shapefile are skipped and a message is logged.
• OmniSciDB now returns an error when running the OPTIMIZE command on a view. OPTIMIZE is not supported on views.
• Resolved a crash on binary operations between TINYINT and DECIMAL/NUMERIC types.
• Resolved a crash when attempting to import to a table with TINYINT columns using StreamImporter or StreamInserter.
• Resolved an issue where a LogicalValues query (query over VALUES with no table in the FROM clause) with a sort could crash the server.
• Resolved an issue where SQLImporter failed to import geo datatypes.
• Disabled ALTER TABLE ADD COLUMN on tables with sort_column. This previously caused the server to crash.
• Improved handling of special characters in SQL queries, including quotes.
• Resolved issues where join query results were not updated correctly with table data changes after an UPDATE, DELETE, TRUNCATE, or ALTER command.
• Resolved an issue where SQL CASE statements with Boolean arguments failed to implicitly cast correctly, resulting in a cast failed error message.
• Resolved an issue where multistep queries with a sort could crash the server. These queries are typically dispatched for choropleth charts in Immerse.
• Fixed an issue where a join on a literal or values table could crash the server.
• Resolved an issue where users created prior to the can_login property being added could not log in.
• Resolved an issue where some API calls would not properly reset a users idle session duration, resulting in premature idle session timeouts.
• Fixed an issue where a string dictionary overflow during CREATE TABLE AS SELECT or INSERT INTO AS SELECT could crash the server. Now, an exception is thrown and the query aborts.

OmniSci Render

New Features and Improvements

• Polygon rendering in cached mode has been deprecated. For migration steps for custom clients, see Migrating Vega Code to Dynamic Poly Rendering.
• Increased default render-mem-bytes config setting from 500MB to 1GB.
• Added a new config option, max-concurrent-render-sessions, to set the maximum number of concurrent render session caches. Default value is 500.

Fixed Issue

• Resolved a hit-test issue when trying to retrieve data from a column that uses LIKE/ILIKE in an expression.

OmniSci Immerse

New Features and Improvements

• SQL Editor is updated:
  • SQL query formatting is now available, which inserts whitespace for additional readability.
  • You can run a selected portion of the SQL query. Selecting text transforms the “Run query” button to “Run selected”.
  • The table browser supports filtering on column type.
  • From the table browser, clicking on a table or column name shows options for inserting “query snippets,” or a shortcut to common queries that provide additional information about the table and its columns.
  • The query history tab lists previously run queries from the active session, and clicking a query displays previously fetched results. Click the insert icon to insert the query back into the input window to re-run it.
  • You can press tab for autocomplete on queries and resize the results window.
• Using multiple databases in Immerse has been enhanced:
  • Dashboard URLs are now unique across databases.
  • You can access additional databases from the account dropdown in Immerse, either in the current browser tab or as a new tab.
  • You can access multiple databases at the same time, as long as the user sessions are active.
• Load time for the dashboards list is reduced significantly.
• Count of records can now be used as filtering criteria during cohort creation.
• Error handling is improved.
• Custom logos can now be configured separately for light and dark themes.
• Logos can be configured to deep-link to a specific URL.

Fixed Issues

• Fixed formatting issue where values above 100M were rounded to two significant digits; additional precision is added to the default number formats.
• Fixed issue on the histogram chart where the last bin was missing. When you log in one browser tab, you are logged out across all other browser tabs.
• Added tooltips in column section to make reading long column names easier.
• Fixed issue with custom measures created with CASE statements not rendering correctly in choropleths.
• Fixed issue where shapefile imports failed on files with uppercase extensions.
• Fixed issue where appending data errored out when import settings were changed.
• In heatmaps, dates are now sorted correctly--by time--instead of alphabetically.
• Upgrading from HTTP to HTTPS now works correctly.

5.1.2 - March 2, 2020

OmniSci release 5.1.2 contains new features and several fixed issues.

OmniSciDB

New Features and Improvements

• Added support for SQL VALUES syntax, allowing literals to be used inline in SQL query expressions.
• Added calcite-service-timeout to the OmniSciDB server configuration parameters, allowing the timeout between OmniSciDB and the Calcite process to be increased for high utilization systems. See calcite-service-timeout on the Configuration Parameters page.

Fixed Issues

• Resolved an issue where multiple, concurrent shapefile imports from Immerse could fail.
• Resolved an issue where concurrent imports and render queries could cause a crash.
• Fixed ALTER TABLE ADD COLUMN in the presence of sharded tables. Previously, the new column was not added to all shards.
• Removed a dependency on nvcc being available in the user’s PATH when compiling C/C++ UDFs. CUDA is still required, but nvcc does not need to be accessible to OmniSciDB.
• Removed GPL-licensed com.unboundid.unboundid-ldapsdk package that had been included in mapd.jar.

OmniSci Render

Fixed Issue

• Added support for transient string dictionary columns in ordinal scales.

OmniSci Immerse

New Features and Improvements

• Added the ability for polygon borders to be the same color as the polygon fill color, rather than always defaulting to white.
• Added the ability to display a list of databases to choose from at login, configured via servers.json. See login_panel_databases.

5.1.1 - February 18, 2020

OmniSci release 5.1.1 contains one significant fixed issue.

OmniSciDB

Fixed Issue

• Resolves an issue with OmniSciDB 5.1.0 on several cloud service provider marketplaces.

5.1.0 - February 5, 2020

OmniSci release 5.1.0 contains many new features and fixes.

OmniSciDB

New Features and Improvements

• Added an option to the web server which allows users to disable TLS certificate verification on startup. While this is not recommended for production deployments, services running in unencrypted mode in a container require this option to properly start using the startomnisci script. The web server backendUrl parameter now also respects https, if configured in the OmniSciDB config.
• Enhanced OmniSciDB log files to include inbound client IP address and protocol for all endpoints and additional logging for import queries when CUDA contexts fail to be created.
• Improved error handling for View queries. Errors now display the view DDL and instructions for resolving the underlying issue that prevents the view from being executed.
• Improved support for EXISTS and NOT EXISTS subqueries.
• Added support for LINESTRING, POLYGON, and MULTIPOLYGON in user-defined functions.
• Improved logging of SAML and LDAP roles and associated actions when authenticating with an identity provider. Logs are written to the INFO log when a user establishes a new session.
• Added support for the can_login user attribute. When set to false, the user is unable to log in to any database.
• Improved performance for multi-column GROUP BY queries, as well as single column group by queries with high cardinality. Performance improvement varies depending on data volume and available hardware, but most use cases can expect a 1.5 to 2x performance increase over OmniSciDB 5.0.
• CREATE TABLE AS SELECT queries now support an optional IF NOT EXISTS clause. For example:

  CREATE TABLE IF NOT EXISTS test AS SELECT. . .

• Improved the resiliency of query parse errors, including allowing for more special characters inside quoted strings.
• Added support for UPDATE via join with a subquery in the WHERE clause.
• Added support for DELETE via join queries.
• Added support for CREATE TEMPORARY TABLE.
• Added support for JDBC authentication using a PKI certificate.
• Added support for signed SSL certificates with a non-system certificate authority (-ssl-trust-ca).

Fixed Issues

• Resolved an issue where embedded tabs or new lines could cause a crash during query parse.
• Resolved an issue where MIN/MAX/SUM over a NOT NULL column could return the NULL sentinel representation instead of NULL when no rows passed the given filter.
• Resolved an issue where SSL keystore passwords were visible in the system process list when OmniSciDB was running.
• Resolved an issue where adding a NULL to a date or timestamp value incorrectly resulted in an addition. Adding a NULL to a date or timestamp now generates an exception.
• Suppressed extraneous log messages when the query completion hints system was activated (typically either by hitting the tab key in `omnisql` or by using the SQL Editor in Immerse).
• Fixed an issue where the GRANT and REVOKE command for dashboard privileges failed when running in distributed mode.
• Resolved various issues when updating variable length datatypes (arrays) containing NULL values.
• Resolved an issue where overflow / underflow errors were not detected during import or cast operations on TIMESTAMP types.
• Improved the stability of JOIN queries in distributed mode.
• Resolved an issue where INSERT INTO TABLE AS SELECT using geospatial columns could incorrectly throw a "column encoding sizes do not match" error.
• Resolved an issue where connections made to the OmniSciDB HTTP(S) endpoint would be refused when running warmup queries.
• Resolved an issue where the log_severity configuration file option failed to read quoted values.
• Resolved an issue where taking the absolute value of a TINYINT column type using the ABS() function could crash the server.
• Improved type checking when comparing geospatial types. Now, all geospatial types are required to have the same SRID, if specified.

OmniSci Render

Fixed Issues

• Resolved an issue where hit-testing results might be incorrect when rendering cached polygons.
• Resolved an issue where hit-testing an accumulation render in distributed mode would return an empty result when using multi-gpu-configured leaves.
• Resolved an issue where users attempting to render the result of a view were required to have access to the underlying table. Now, only access to the view is required for render query execution.
• Resolved an issue where accumulation renders could crash in distributed mode when the results are relatively unbalanced across the leaves.
• Resolved an issue where an incorrect buffer layout error could be thrown incorrectly when using Vega transforms in successive renders.

OmniSci Immerse

New Features and Improvements

• Log-ins are fully sessionized and persist across page refreshes.
• Pie chart now supports “All Others” and percentage labels.
• Cohorts can now be built with aggregation-based filters.
• New filter sets can be created through duplicating existing filter sets.
• Dashboard URLs now link to individual filter sets.
• The top level "toggle off" filters option now supports toggling those filters back on.
• A 5-second refresh rate has been added as the lowest refresh setting for dashboards.
• Added ability to increase the size of bubbles on the Bubble chart, and to be able to manually configure the MIN and MAX size of bubbles when there is a size measure.
• Improved display and messaging of error modals.
• Added a label for the current database in the account settings dropdown.

Fixed Issues

• Fixed error handling resulting in popup loop due to incorrect Custom Measure syntax, on choropleth rendering, and during specific cases of chart type switching.
• Fixed error handling where there was no error information presented when a dashboard name was duplicated.
• Fixed an issue where after adding a color dimension on one source, the user is unable to change other sources to the bar style.
• Fixed an issue where changes to global filters were not picked up by the Save button.
• Fixed an issue where there was a shape filter delay on the zoom in/out button action.
• Fixed an issue where Bar chart inverse filters did not correctly account for NULLs.
• Fixed an issue where Bar and Table charts were showing errors when switching to charts with a color dimension.
• Fixed an issue in Bubble chart where switching to an extract unit caused Immerse to crash.
• Fixed issues related to dark and light theme font color inconsistencies.
• Fixed an issue where chart filters on extracted timestamps appeared incorrectly in the filter panel when time series charts were brushed.
• Fixed a formatting issue causing x-axis labels to only show two significant digits.
• Fixed an issue where certain filters from charts were not showing up in the filter panel.
• Fixed an issue where filtering on a measures only Table chart caused Immerse to be in an error state.

5.0.1 - November 18, 2019

OmniSci release 5.0.1 contains two critical fixes.

• Fixed an issue where CREATE TABLE AS SELECT or INSERT INTO TABLE AS SELECT could return incorrect results in distributed mode.
• Fixed an issue where projection queries over sharded tables could return incorrect results in GPU mode.

5.0 - November 14, 2019

OmniSci release 5.0 contains many new features and fixes.

OmniSciDB

New Features and Improvements

• OmniSciDB now supports certain forms of correlated subqueries.
• OmniSciDB now supports update via subquery, to allow for updating a table based on calculations performed on another table.
• Added support for binary dump/restore of database tables.
• Added support for compile-time registered user-defined functions in C++, and experimental support for runtime user-defined SQL functions and table functions in Python via the Remote Backend Compiler (see omnisci_examples folder within the OmniSci JupyterLab instance for example UDF/UDTFs).
• Multistep queries that generate large, intermediate result sets now execute up to 2.5 times faster by leveraging new JIT code generator for reductions and optimized columnarization of intermediate query results.
• Added support for LIMIT 0.
• Added SAML support for auth0 and ping, and Oracle Access Manager.
• Added support for window functions inside complex expressions.
• Standardized encrypted transport options across ODBC, JDBC, SQLImporter, and omnisql.
• Added an option to log LLVM IR or NVIDIA PTX to a log file per query-step using --log-channels.
• Added support for geospatial POINT construction from literal or computed values.
• Added support for SQL operators IS TRUE and IS NOT TRUE.
• Added support for SQL operators EXISTS and NOT EXISTS.
• Added support for geospatial types to StreamImporter.
• Added support for updates through joins.
• Docker editions now support Ubuntu 18.04.
• Generalized performance and optimization improvements.

Fixed Issues

• Aggregating empty results in distributed mode could cause a crash.
• The Apache Arrow interface (sql_execute_df) properly handles empty results.
• Fixed an issue where join queries could give no results in distributed mode, even if the tables were properly sharded or replicated.
• Fixed an issue where time zones could be parsed incorrectly on import of date/time/timestamp types.
• Improved clarity of the get_db_objects_for_grantee() (Thrift) / \object_privileges (omnisql) APIs and resolved an issue where all privileges were not always included.
• Fixed an issue where Calcite could crash on malformed queries.
• Resolved issues where aggregate functions could overflow, returning an incorrect result.
• Resolved an issue where the S3 importer could crash when attempting to import from a SSL encrypted OmniSciDB server.
• Resolved an issue where small deviations in image appearance could occur during anti aliasing.
• Resolved an issue where adding a column to a sharded table could corrupt the table.
• Resolved an issue where commas in log messages could cause the logs to be improperly parsed by OmniSciDB or third-party utilities.
• Resolved an issue where the ARRAY constructor could crash in the presence of a sort or limit.
• Fixed an issue importing MULTIPOLYGON columns from PostGIS via SQL Importer.
• Resolved an issue where NULL was not handled correctly in AND/OR operations.
• Resolved an issue where creating a new table via CTAS would fail with a parse error if the table name contained the word with.
• Resolved an issue where CTAS would not properly apply the LIMIT or OFFSET if present in the query.
• Resolved an issue where DECIMAL columns could overflow when used in a CTAS query.
• Fixed a crash that could occur when a user reauthenticates after session timeout.
• Fixed a crash when filtering geospatial columns by IS NULL/IS NOT NULL. (Geospatial nullability is not yet supported.)
• Resolved an issue where compressed POINTs could be mishandled during result set iteration.
• Addressed known crashes in aggregators with variable-length arguments, including various geometry types and nonencoded strings.
• Fixed an issue during OmniSci ODBC Driver DSN Setup, where the “Ignore Invalid Certificates (not recommended)” field is unavailable in the dialog box.
• Handle more gracefully a Calcite exception error that caused the server to abort.
• Fixed a crash occurring when a prior query result was passed through a SQL function using a cast.
• Resolved a crash that could occur when running aggregate functions on geospatial types in distributed mode.
• Fixed an issue where binary operations on TINYINT types could crash.
• LDAP and SAML role synchronization is now case insensitive, matching OmniSciDB internal representation.
• Fixed an issue where combining columnar support with a shared table under ORDER BY or LIMIT could return an incorrect result.
• Fixed an issue where a long query would time out a session, invalidating the operation or series of operations that required a longer execution time.
• Resolved an issue where a table could be created with a specified shard_key but without a shard_count.
• Fixed an issue where the AVG aggregate function is not properly cast when passing floating-point data.
• Fixed an issue where SELECT queries with no filter and a LIMIT could return incorrect results on a table where one or more rows have been deleted.
• Fixed an issue where a JOIN may not return correct results on a distributed setup.
• Resolved issue where embedded new line characters could cause CTAS or ITAS queries to be rejected.

OmniSci Render

New Features and Improvements

• SPIRV is now the default shader IR, and the --enable-spirv flag has been removed. Fallback to disable SPIRV is supported automatically.
• The angle (rotation in degrees & radians) property is now available in Vega symbol marks.
• The wedge property is now available in Vega symbol shapes.

Fixed Issues

• Fixed a choropleth rendering issue when a FLOAT column is used to drive the color.
• Fixed a data underflow issue for integer inputs for quantitative/qualitative scales.
• Addressed possible validation issue when a floating-point column is used as an input for scales with explicit integral values defined for the domain.

OmniSci Immerse

New Features and Improvements

• New filter panel that delivers major improvements, including:
  • Filter Sets providing access to different sets of filters in one dashboard.
  • The ability to toggle filters on and off, in addition to removing them.
  • The ability to visualize and modify cross-filters directly from the filter panel.
  • The ability to visualize from which charts cross-filters are generated.
  • The ability to filter columns by type, making it easier to find the correct column for a filter.
  • A custom filter builder that gives visual access to table columns for greater context and easier generation of filters
  • The ability to filter a text column on whether it starts with or end withs a certain pattern.
  • The ability to multi-select patterns on which to filter a text column.
  • Cohort building and analysis capabilities, including:
    • The ability to use global and cross-filters to interactively build cohorts of interest.
    • The ability to apply a cohort as a dashboard filter, either within the existing filter set or in a new filter set.
• Data Catalog, a repository of datasets that users can use to enhance existing analyses.
• Frontend-rendered choropleths now support the selection of base map layers.
• Duplicate queries in charts were significantly reduced for performance improvements.
• Map scales have been added to map charts.
• Dark mode is the default Immerse theme.

Fixed Issues

• Fixed issue in multilayer map charts where prior filters were being kept on the dashboard, even after the layer was removed.
• Fixed an issue where color legend min/max inputs were not retained.
• Fixed an issue where table name changes were not saved in the Update Settings flow.
• Fixed experience in combo chart where a chart with all negative values had its axis inverted to positive; now the values stay in the original orientation.
• Fixed an issue in map chart where resizing the chart would create an asymmetric resize animation.
• Fixed an issue in dashboards search where results do not display correctly if only one character is used for the search.
• Fixed an issue where geo join columns can be used in custom color measures across both the fact and geo join tables.
• Fixed an issue in scatter chart where the circle filter goes out of container after clicking the cancel button.
• Fixed an issue where global filters errored with date inputs that returned no results.
• Fixed an issue for combo chart where secondary measure formatting was incorrect.
• Fixed an issue where linemap size measure had incorrect range values.
• Fixed an issue where “OmniSci” appears in browser tab before being overwritten via whitelabeling.
• Fixed an issue where line chart series were not fully migrated to combo chart.
• Fixed an issue where Oracle Access Manager deep links were working incorrectly.
• Fixed an issue in combo chart where bars switched to the secondary axis overlap the bars in the primary axis.
• Fixed an issue where the streets Immerse theme appeared layered on top of points.
• Fixed a lag in shape filters.
• Fixed an issue where invalid custom SQL filters would create a dashboard-level error state.
• Fixed an issue where number charts were not updating with the rest of the dashboard.
• Fixed how web links in text charts were incorrectly deep-linking outside the product.
• Fixed an issue where custom filters containing OR statements were incorrectly applied to queries sent to OmniSciDB from Immerse.
• Fixed an issue where pointmap values would disappear after the map theme was changed.
• Fixed an issue where changing chart type from line to combo would remove locked x-axis values.
• Immerse chart type selections scroll in a width-constrained browser.
• Import/export dashboard metadata icons have been updated for improved clarity.

4.8.1 - August 27, 2019

OmniSci release 4.8.1 contains several fixes.

OmniSciDB

Fixed Issues

• Fixed an issue that occurred when creating a table with an invalid shared dictionary column.
• Resolved an issue where a CSV import with a row delimiter inside a quoted string would cause a premature line break, resulting in incorrect imported values for all subsequent rows.
• Resolved an issue starting OmniSciDB inside a Docker image on legacy versions of Docker (affects version 18.06 and prior).
• Fixed an issue affecting importing data from AWS S3 when the import statement was malformed.
• Fixed an issue where importing geospatial data using the SQLImporter would fail after importing 10,000 records.
• Fixed an issue affecting multi-step distributed queries over empty tables.

4.8.0 - August 6, 2019

OmniSci release 4.8.0 contains several new features and fixes.

OmniSciDB

Major Changes and Known Limitations

• Jupyter notebook integration is an experimental feature. See Jupyter Integration.
• Thrift methods get_frontend_view, get_frontend_views, create_frontend_view, delete_frontend_view, get_table_descriptor, and get_row_descriptor are deprecated and no longer function. If you use one of these methods, you receive an error message describing how to migrate to the new API calls.
• Arrow version upgraded to 0.13.0.
• Minimum supported Nvidia CUDA driver is now 410.48.
• Significant additional controls available in the logging framework. See Logs and Monitoring. Note that the individual file names for the logs have changed from <service_name>.<machine_name>.<severity>.<timestamp> to <service_name>.<severity>.<timestamp>.log.
• When running an encrypted server configuration in Docker containers there is a current limitation that the aggregator omnisci_server and the omnisci_web_server must be running in the same Docker container and they cannot be started with the startomnisci script, which is the default. The use of a Docker COMMAND is required to start the services. For example:

  command: |
  /bin/bash -c "/omnisci/bin/omnisci_web_server --config /omnisci-storage/omnisci.conf & /omnisci/bin/omnisci_server --config /omnisci-storage/omnisci.conf

  For more information, contact OmniSci support.

New Features and Improvements

• Added support to log user's roles information during login event.
• Added support to allow communication among distributed system components to use encrypted binary connections.
• Added OPTIMIZE support for sharded tables, allowing administrators to create more optimal schemas.
• Added support for ST_Point geo constructor.
• Added new flag is_super (whether current session is super user) to session info returned from Thrift get_session_info.
• Improved performance of some high-cardinality group-by queries.
• Improved performance of ST_Contains by adjusting the bounding box heuristic.
• Improved error messages for unsupported function/type combinations.

Fixed Issues

• Fixed issue with ODBC connector not working with PowerBI.
• Fixed segfault during COPY FROM direct to leaf node on distributed setups.
• Fixed issue with Check Failure related to a query on a distributed sharded configuration.
• Improved signal handling and shutdown procedures for OmniSciDB server.
• Fixed an issue in geo join due to table join reordering.
• Fixed an issue where a non-Admin user could set a license key.
• Fixed an error resulting from running the same Vega render twice in quick succession.
• Fixed issue with a program error from a query that casts a TIMESTAMP(0) to TIMESTAMP(3).
• Added support for more graceful shutdown of OmniSciDB when receiving OS signals.
• Fixed issue with web server incorrectly switching to SAML mode if a session ID cookie is present.
• Fixed issue for server program errors on GROUP BY queries with nullable SMALLINT and TINYINT keys.
• Fixed issue where empty string as group name causes server program error.
• Fixed issue where the startomnisci script no longer works on CentOS.
• Fixed issue where ROUND_TO_DIGIT(x, y) (all data types) resulted in a server program error.
• Fixed edge case where Log10(NULL)/ Log(NULL)/LN(NULL)/EXP(NULL) was being computed and resulted in a number.
• Fixed issue where rendering join queries with multiple rowids produced the wrong results.
• Fixed issue where \detect did not appear in \h listing in omnisql.
• Fixed issue where requesting Arrow results on CPU (via select_ipc) results in the query executing on CPU, rather than just the resultset being located in CPU memory.
• Fixed issue to workaround where cuDF does not recognize arrow::date32 type, using arrow::date64 for gpu buffers.
• Fixed issue where query returns different results on distributed and single-node.
• Fixed issue where hit testing on Line map with color measure selection throws an error.
• Fixed issue with a render resulting in CHECK failure from a Choropleth Vega query.
• Exception for a query and subquery with differing limits.

Immerse

Major Changes

• Line charts are deprecated in 4.8. While existing Line charts continue to function normally on dashboards, new edits to those charts cannot be saved. OmniSci recommends upgrading Line charts to Combo charts to take advantage of the latest feature improvements.
• Over the coming releases, OmniSci will stop developing new Immerse features for Internet Explorer. We strongly recommend all customers switch to a non-IE browser to experience the latest Immerse features.

New Features and Improvements

• Immerse now directly integrates with JupyterLab. You can send SQL queries to Jupyter from SQL Editor, or access it directly from top level navigation. See Jupyter Integration.
• Combo chart supports focus zoom. Moving the mouse wheel up and down zooms the chart in/out, while holding an Option or Shift key while scrolling pans the chart from right to left. Clicking the Reset button on hover resets the chart to the original timeframe.
• You can visualize Immerse in a new Dark Mode theme, accessible through account settings.
• Immerse supports importing and visualizing millisecond timestamp data.
• Immerse lets you duplicate charts within dashboards.
• Immerse lets you export and import dashboard metadata within Immerse.
• Immerse now pops up a modal when you are using a non-WebGL supported browser, informing you of the reduced experience.
• + and - buttons appear on geospatial charts as zoom controls.
• Offline maps now include a Minimalist base map layer at the country level.
• Custom base map layers are now defaulted first for easier selection.

Fixed Issues

• Updated combo chart's dynamic top 5 colors to consistently show the five series with the most data volume.
• Fixed issue where pop-ups on Choropleth were failing to appear for views created from multiple data sources.
• Fixed custom colors resulting in an add/remove loop in Combo chart, and added a toggle to control whether the chart is in dynamic or fixed mode.
• Incorrect global filters now error gracefully rather than remain part of the active filters list.
• Fixed issue where Choropleth was representing the Other category always as grey; now the Other category correctly shows the selected color.
• Fixed issue where error messages were showing up during dashboard duplication.
• Fixed issue where key manager shows key issuer instead of client name/company.
• Fixed issue where case-different, same text columns were causing a "table does not exist" error.
• Fixed issue where mapd-connector running under Node.js was incorrectly returning null for some functions.
• Fixed issue causing blank page when Add Chart button is pressed during dashboard load.
• Fixed issue where raster chart gaps showed when a Combo chart was present on the dashboard.
• Fixed issue where x-axis scale on a Combo chart is incorrect when the relative date filter is applied in addition to a second measure.
• Fixed issue where Bar chart was incorrectly applying formatting to color measures and dimensions.
• Fixed issue where filter label and dropdown menu were unconnected during the scroll action.
• Fixed issue where Combo chart binning label was incorrect.
• Fixed issue where certain custom colors were not discoverable in the Combo chart search dropdown.

4.7.1 - June 27, 2019

OmniSci release 4.7.1 contains several fixes.

OmniSciDB

OmniSci Immerse

• Fixed an issue causing an increased margin on the right side of geospatial charts.
• Fixed an issue causing errors during dashboard load.

4.7.0 - June 13, 2019

OmniSci release 4.7.0 contains several new features and fixes.

See also the OmniSci Blog.

Major Changes and Known Limitations

• OmniSci Core has been renamed to OmniSciDB throughout the product and in the documentation.
• Thrift methods get_frontend_view and create_frontend_view are deprecated.
• Default user and database names are changed. Note and update your scripts accordingly:
  • The default username has been changed from mapd to admin. All new data directories created with initdb create a superuser with the username admin. Existing usernames are not migrated.
  • The default username for all utilities is now admin. Existing users must override the default username in connection strings / command-line arguments of utilities (for example, omnisql --user mapd).
  • ALTER USER <> RENAME TO <> is now supported, allowing you to rename the existing mapd user to admin if you do not want to use username overrides in your applications.
  • After rename, any sessions connected to the renamed database are invalidated and need to log in again.
• The default database has changed from mapd to omnisci. All new data directories create a default database named omnisci instead of mapd.
  • The default database for all utilities is now named omnisci. Similar to the username change, you can override the default database: for example, omnisql --db mapd for existing installations.
  • ALTER DATABASE <old_db> RENAME TO <new_db> is now supported, allowing you to rename an existing mapd database to omnisci for convenience.
  • After rename, any sessions connected to the renamed database are invalidated and must log in again.
• OmniSci 4.7 fixes an issue with OPTIMIZE TABLE WITH VACUUM, which previously could cause an incorrect entry in the storage for the optimized table. If you have used OPTIMIZE TABLE WITH VACUUM, you should export and then re-import or reload your data.
• OmniSci 4.7 is the last version to support NVIDIA driver version 386. Future releases will require NVIDIA driver version 410 or higher.
• Any blank passwords (that is, nothing) are reset to a random 72-character string. This is required to prevent any accidental blank passwords enabled in the system. Previous versions automatically created blank passwords when logging on with LDAP or SAML, which creates a conflict with the new fallback to local login support.
• ODBC usage requires the license on the connecting server to be Enterprise enabled.
• \clear_gpu and \clear_cpu now require superuser privileges to run.
• Users that have access to a database through a role are now returned as part of a get_users call for a specific database. More users are now visible in shared Immerse dashboard dialogues.
• Apache Calcite version is now 1.19.
• For geojoin-configured Choropleth charts, Immerse now generates a new poly render query that no longer uses the server poly cache. The poly cache may be deprecated completely in a future release. The change includes better distribution of poly data across all available GPUs, causing an overall memory footprint improvement.
• TEXT ENCODING DICT(n) columns cannot be encoded as less than 32 if they are to be used as a SHARD KEY on a sharded table. This restriction will be addressed in a later release.

OmniSciDB

New Features

• OmniSciDB now supports INSERT TABLE AS SELECT (ITAS) on both single node and distributed, including sharding support.
• Joins on bucketed date/time are now bucket-size aware and more memory-efficient.
• OmniSciDB now supports Window functions. Empty Window function partitions are currently not supported; that is, all Window functions must include a PARTITION BY key in the OVER clause. Also, OmniSci recommends that you use Window functions over aggregate query outputs at this time (instead of raw data). Future releases aim to improve scale and performance and remove these restrictions; see the documentation for details on what is supported.
• The default username is now admin.
• OmniSci DB now supports ALTER USER RENAME, which allows you to rename users.
• OmniSciDB now supports Domain-based Authentication, allowing for multiple login modes (SAML, LDAP, and local) depending on user ID. This allows you to create OmniSci admin accounts outside of LDAP and SAML. The server first checks for SAML and then for LDAP. You configure this in the server configuration file. In addition, the server now allows authorization to fall back to local OmniSci authorization if LDAP or SAML (or both) are enabled, but fail. The --allow-local-auth-fallback controls this behavior, and is set to false by default.
• OmniSciDB now supports Distributed Polygon Rendering, allowing for much greater scale in rendering. For Enterprise customers running in Distributed mode, the Choropleth chart in Immerse now supports backend rendering by default.
• ODBC allows for cookies on http and https connections; relevant only if a load balancer is used.
• OPTIMIZE TABLE now recomputes metadata for dictionary-encoded string columns, supporting updates and deletes on dictionary-encoded string data.
• Performance improvements:
  • (Experimental) Sorting columns at import time (a user operation) can result in improved performance by up to 3.5x on query patterns that involve filters on those columns.
  • Improvements in distributed query performance involving joins. Distributed execution mode requires that your tables are structured correctly for optimal query execution. Immerse queries are formulated to work in Distributed mode, but customers using OmniSciDB with custom applications need to ensure that any table on the inside of a join is either replicated or the join relation is on a sharded column in both tables, unless both sides of the join relation are the result of an intermediate query step.
  • View optimization is now on by default. OmnisciDB optimizes all view queries through RA rewrite. This should result in a noticeable reduction in the number of steps required by many view queries.
  • Improved performance in distributed mode from more efficient result set serialization.
  • Improved performance of multistep queries in which subsequent steps are projection nodes.
  • Improved performance of filter queries in which the filtered column comes from the inner table in an inner join relation.
• Added support for SQL WITH syntax in poly render queries.
• Added support for IF NOT EXISTS and IF EXISTS to database DROP and CREATE statements, respectively.
• Added new logging messages for monitoring query execution in Distributed mode.

Fixed Issues

• Fixed SAML-configured login that made regular access to other utilities awkward or restrictive.
• Improved performance by eliminating runtime loads of fixed-length array sizes that slow down POINT geo functions.
• Added cookie support to ODBC driver.
• Fixed a render error in Vega runtime that dumped the entire FRAGMENT SHADER code in the log.
• Fixed an issue where a scalar sub-query returned multiple rows exception was thrown while validating scalar sub-queries.
• Fixed an issue where update storage layer erroneously used std::numeric_limits< double >::min() when initializing the max range of a nonnegative number.
• Fixed a program error resulting from a multi-level join query.
• Fixed a program error on hash tables with more than 2B entries.
• Fixed an issue where a cast from BOOLEAN to BOOLEAN was not supported, resulting in an error.
• Fixed a program error resulting from tables with a negative epoch.
• Fixed an issue where the importer did not skip rows with incorrectly sized fixed-length arrays.
• Fixed an issue where an exception resulted when \clear_gpu was run after any render.
• License configuration parameter now allows quotes.
• Fixed an issue resulting in incorrect results on CPU for group by query with CASE.
• Fixed a program error that resulted from a COPY FROM on Parquet with POINT geo type.
• Fixed a render error when going from a non-in-situ query to a projection query that uses multiple GPUs.
• Fixed a render error when updating from min/max -> stddev density accum renders.
• Fixed an issue where shard count requested with CREATE was different from what was reported with \d in Distributed mode.
• Fixed an issue where lack of rowid in projection queries negatively impacted Vega property updates.
• Fixed program error when shard count exists, but shard key does not.
• Fixed an issue with distributed line rendering not respecting Vega line format with scalar columns.
• Fixed an issue involving joins on a value or null. For example, a.x = b.x or a.x is null and b.x is null.
• Fixed a regression with multilayer render on Distributed systems using Vega xforms and aggregation/projection queries.
• Fixed importer program error resulting from shapefile with null geo features.
• Fixed program error during SAMPLE on a geo join in nondistributed mode.
• Fixed program error with GROUP BY with BIGINT and INT columns.
• Fixed server program error resulting from CASE WHEN/THEN string test select join with INT, FLOAT, and SMALLINT columns caused a server check and program error.
• Fixed an issue where a user could not be dropped if the username contained a period.
• Fixed program error with geo function filters under certain circumstances.
• Fixed an issue with JOIN WITH subquery not returning results in a Distributed environment.
• Fixed an issue where a non-superuser user was not included in get_users Thrift call when the user has ACCESS/SELECT database rights.
• Setting a license key through Immerse or omnisql now works in HA mode.
• Invalid SQL casts now produce more meaningful error messages.
• Fixed a program error involving render query hit testing in Distributed mode.
• Fixed a Vega bug involving dictionary-encoded string columns from an aggregate target in an ordinal scale.
• Improved performance of projecting queries with geospatial targets in Distributed mode.
• Calcite server no longer has a program error if the user passes in an empty string.

OmniSci Immerse

Known Limitations

• Distributed Poly rendering on Choropleth charts introduced in 4.7 has the following known limitations (Enterprise Edition):
  • With the server-side poly cache no longer being used, some render performance issues with geojoin configured Choropleth charts can occur. If the dimension table for the geojoin (that is, the poly table) has a large number of rows, you might see some performance degradation in release 4.7. Internally, tables <100K polygons tend to perform equally to the cache, 100K-500K rows sees mild degradation, and >500K rows can see more substantial degradation. Performance is ultimately dependent on the total number of vertices (not rows).
  • Currently, neither Immerse nor the server can be configured to use the poly cache. This limitation impacts Immerse users only. Custom clients doing server-side polygon rendering using join queries can continue using existing Vega code and use the poly cache. OmniSci recommends that you switch to the new query format; instructions for doing so will be included in a future release. If you prefer to make these changes beforehand, contact OmniSci support.
• A known issue with Distributed poly rendering and replicated poly tables can result in overdraw of the polygons from each leaf in projection queries. This might not impact the final render if the polygons are rendered fully opaque. If they are rendered with transparency, the final render is less transparent than expected due to overdraw.
• Line chart in Immerse is scheduled for deprecation in a future release, to be replaced by the Combination Bar/Line (Combo) chart.
• For geospatial charts, changes in release 4.7 introduce an increased margin on the right side. This is scheduled to be fixed in a future release. As a workaround, resize the window to remove the margin.
• You must upgrade to Windows 10 1903 and Edge 44.18362 to use Edge with OmniSci. If you choose not to upgrade, you can use an alternate browser with OmniSci.

New Features

• Visual Data Fusion supports plotting data from multiple data sources on the same Combo chart, which also enables cross-filtering with other charts. This was previously available only on geospatial charts.
• Filtering on an aggregated value is supported in Pointmap and Scatter Plot charts.
• Data import status provides time estimation and information on import success.
• Server-rendered Choropleth charts are supported in Distributed mode.
• Data import popovers provide help text on file formats available for import.
• Binning limits on Line and Combo charts are increased from 400 bins to 1000 bins.

  Note	When binning granularity is set to auto mode on a Line or Combo chart created prior to version 4.7, the reported bin granularity on the chart (for example, Hour) might not match the actual bin value (for example, Day). Workaround: Recreate the chart in version 4.7.

Fixed Issues

• Fixed an issue for applying a global timestamp filter to a Combo chart where the displayed dimension is not of TIMESTAMP data type.
• Fixed an issue where user did not appear in list for sharing despite having database permissions.
• Fixed issues with Combo chart related to setting min/max values on the x and y axes.
• Fixed an issue where the locked icon ona Combo chart disappeared, despite having a fixed value set on the axis.
• Fixed an issue where the top bin of binned data was missing values due to rounding discrepancies.
• Fixed an issue where Combo chart did not bin correctly on cross-filter.
• Fixed an issue where a new license could not be pasted in after an previouis license expired.
• Fixed an issue in Bar chart where sorting by measure did not update the chart.
• Fixed an issue to better address auto-binning in Combo chart.
• Fixed an issue in Stacked Bar chart for setting custom colors.
• Fixed an issue in Line chart for setting an incorrect aggregation due to color dimension.
• Fixed an issue in Combo chart for automatic axis labels.
• Fixed issues with Choropleth chart error handling.
• Fixed chart-switching error issues in Pointmaps.
• Removed Append Data button for users without INSERT permission.
• Fixed x-axis label issue in Stacked Bar charts.
• Fixed tooltip sorting in Combo charts so that the ordering is consistent.
• Removed default brush on focus chart after a range chart brush is set.
• Fixed an issue causing an error state for Append Data using Internet Explorer 11.

4.6.1 - April 29, 2019

OmniSci release 4.6.1 contains several fixes.

OmniSci Core

Fixed Issues

• Fixed an issue where a grouped pointmap with multiple "SAMPLE" measures could cause a program error.
• Fixed an issue where a JOIN on the result of an unnest operation caused a program error.
• Fixed an issue where QueryMemoryInitializer was not passed safely to the JOIN loop builder.
• Fixed an issue where queries retried on CPU with subqueries that had already been executed would cause a program error.
• Removed buffer size limitations between cluster aggregator and leaves.
• Fixed an issue where sql_execute_df would throw an exception when encountering a BOOL column.
• Restored SQL execute timings to std_log output.
• Fixed an issue with intermittent server program errors with CUDA error code 201 in egl compositor.
• Improved error resolution when an error is thrown during the build of a poly cache.
• Fixed an issue with COPY TO behavior when executing in HA mode.

SELECT TIMESTAMP '2016-02-28 1:23:45' + INTERVAL '24' HOUR
SELECT TIMESTAMPADD(HOUR, 24, TIMESTAMP '2016-02-28 1:23:45')