SQL Server query optimization for Dynamics ERP

SQL Server query optimization for Dynamics ERP

SQL Server query optimization for Dynamics ERP


1. Why perform SQL server query optimization for Dynamics ERP systems like Business Central or NAV?

In recent years, you've seen incredible improvements in many areas of database servers:

  • Database optimizers are constantly improving and finding ways to make queries more adaptable and overcome areas of poor optimization.
  • Memory speeds have increased massively due to advances in both memory technology and network bandwidth, CPUs have become much faster, and memory prices have dropped dramatically.

However, there are still professionals (like us) who make their living tuning queries and training others to optimize queries. This process involves finding certain slow queries (long durations) that are critical to Dynamics Business Central or NAV performance. These "slow SQL queries" can result from making strategic changes, whether in code, database structure, NAV server instance configuration, or something else. Optimizations are often necessary to ensure that these specific SQL queries execute consistently at a certain speed or meet a required performance standard.
But so why SQL server query optimization despite all the improvements?

Very simple:

  • Data volumes are also increasing dramatically
  • Customer/user expectations for performance/speed of applications like Dynamics Business Central have also increased
  • Customers/users expect not to wait for results now - they want to see current status immediately

2. Who should perform the SQL query optimization?

SQL query optimization is performed by:

  • Database Administrators
  • C/AL or AL developers who specialize in performance or specifically in databases.

Full-stack developers for AL and C/AL generally do not perform query optimization unless they have a specific interest or work experience. This is more of a specialization than a "quick learn" task, so most full-stack developers simply don't have time and need to hire a more specialized person to help them. Teams that don't have a readily available specialist can periodically bring in consultants like us the DynamicsProject Team to help.

There are also many database administrators who manage SQL databases where only "basic" availability and performance are required. These SQL databases are used by cost-conscious organizations that don't need to tune every database like a "race car": Most of their SQL databases are used by internal users who are used to Dynamics ERP performance and speed moderation.

3. What skills are required in SQL query optimization?

Most of us are not wizards at TSQL, but many DBAs have gotten pretty good at query tuning (e.g., through our V8 Performance Workshops).

A very unscientific estimate of what skills make someone in query optimization:

40% - The time, interest, and resources (including a network of people to ask) to build a comprehensive profile of "performance information" - this is patterns that don't optimize well, boundary conditions where performance becomes poor, understanding trace flags, and configuration on TSQL.

30% - Interest and ability to learn how the database engine optimizes and processes queries using indexes and other resources, and what affects parallelism when running multiple queries simultaneously against a live database.

10% - Understanding of the TSQL language and various ways to rewrite a query in conjunction with Dynamics AL or C/AL programming to produce a specific result set. This often only works with database DevOps.

What is database DevOps?
Let's start with the definition of DevOps. DevOps is a set of practices that combine software development (dev!) and IT operations (ops!) with the goal of delivering more features, fixes, and updates faster, in alignment with business goals. Database DevOps applies the same principles and ensures that AL or C/AL database code is included in the same process as development code.

Database DevOps helps teams further identify and streamline the application development and release process by addressing a known bottleneck: AL or C/AL Dynamics source code changes.

4. SQL query optimization tools and their cooperation.
4.1 Execution plans

How the SQL query from Dynamics Business Central is executed behind the scenes.

  • "Estimated" execution plans show the decisions the optimizer has made to execute the query, including the estimated number of rows that will flow through the different parts of the plan
  • "Actual" execution plans are estimated plans that are updated with runtime statistics, e.g. how many lines have passed through the plan. If the plan is "adaptive", it contains some information about which options were chosen
4.2 Query memory

SQL Server 2016+, all editions

  • This function tracks execution plans and aggregated runtime metrics (duration, CPU usage) along with aggregated wait statistics
  • This also has the possibility to "freeze" plans
  • Query store information is restored with the database itself so that it can be shared between environments as needed.
4.3 Dynamic Management Views and Performance Indicators (DMV)
  • These help to understand the bottlenecks of the whole instance during slow performance
  • Example: Overall maintenance statistics for the instance and metrics on memory latency during the time the queries were poorly executed can help explain whether the query really needs to be optimized
  • Query optimization often requires callbacks for "workload optimization"
  • SQL trace and enhanced event traces (enhanced events are a lightweight performance monitoring system to collect data and are the foundation of V8 Search XE).
  • The "Old" SQL Profiler is difficult to use for query optimization in "live" operations, as they can easily slow down your workload and cause performance issues when tracing.
  • Execution plans (filtering does not help in this case, the plans are all examined / collected and the filter is applied too late).
  • Wait Statistics (filtering can help here, but the collected data is so massive that you have to be very careful - and sorting and querying the collected data is also quite cumbersome
  • "Business Central Server Trace Events" There are two event trace providers that publish different trace events in the event log: Microsoft-DynamicsNAV-Server and Microsoft-DynamicsNAV-Common. The Microsoft-DynamicsNAV-Common provider is for telemetry trace events only. All other events use Microsoft-Dynamics NAV server. Usually, you need to specify the event trace provider in the monitoring tool you use (e.g. V8 Search XE)
  • "SQL Trace Events" trace a specific set of SQL statements executed by the Business Central Server instance against the Business Central database on SQL Server

5. Difficult problems - dispute over resources

It's hard to predict how SQL queries will interact with each other in a live workload

Shared resources:
  • Memory for queries - a certain amount of memory must be allocated for sorting/linking/moving data in a query. Many queries running at the same time that require a lot of memory can cause problems with this. (Sometimes queries assume they need much more of this memory than they need, and it needs to be optimized - this will probably not be aware to users outside of a live workload).
  • The number of queries that make changes and the approach to blocking are difficult to predict outside of a live workload (changes in query plans can cause blocking when they were not previously present)
Changes in server resources - even improvements - can cause blockages if they were not there before
  • Example: Switching to a new server with more memory and faster CPUs resulted in an increase in lock wait times due to reduced in-memory wait times and faster query execution.

6. Is a check in the "live" SQL database required?

Yes. One example of this is parallelism.

Optimizing the degree of parallelism for a workload and for specific queries in that workload is usually quite hardware specific, and you need a live environment.

Workload- „Replays“ sind im Toolkit von SQL-Servern verfügbar, aber sie sind:

  • Repetitions only - you can't meaningfully "boost" the activity (deleting the same lines 10 times is not the same as deleting different lines 10 times)
  • Time-consuming to set up

7. Automated SQL server query optimization: history and development
7.1 Automated plan correction

SQL Server 2017+, Enterprise Edition

  • Built on the query store
  • Detects SQL queries that are sometimes fast and sometimes slow
  • Can only recommend changes if desired and set up
  • Can freeze plans, test to see if it helps, and respond accordingly (Freezing is intended as a temporary solution - it is recommended that a user evaluate the query for optimization as a longer-term solution)
  • Very good function for identifying parameter sniffing
7.2 Intelligent Query Processing
  • "Intelligent Query Processing" (IQP) have been released in the latest versions of SQL Server
  • Several of these features fix common query optimization issues in SQL Server
  • More information: Intelligent Query Processing in SQL Databases.

8. Common errors and pitfalls in SQL Server query optimization.
  • Lack of connection between DBAs and Dynamics development teams
  • Lack of knowledge of the execution plans in the team
  • Lack of understanding of SQL Server isolation levels and "optimistic" options
  • Lack of knowledge of how Dynamics AL or C/AL programming is implemented on the SQL Server

We will be happy to personally answer any further questions you may have on this topic.
Your dynamicsproject.com Team

Identify table locks in Dynamics NAV

Identify table locks in Dynamics NAV

Dynamics NAV Blocking Message


The identification of table locks in Dynamics NAV has completely changed since Dynamics NAV 2013 compared to the Navision Classic Client. The way Dynamics NAV users connect to the SQL database has been restructured by the RTC Client. Users connect to the NAV service tier from the Windows client, web client, Sharepoint client or web services. The new so-called "three-tier architecture" separates database, server and client tiers.
The user account that runs the NAV service tier is the only one that actually connects to the SQL database. There are many good reasons for this:

  • Dynamics NAV Server uses an ADO.NET interface, which is a data access layer that supports SQL Server connection pooling. This simplifies the deployment of the Microsoft Dynamics NAV three-tier architecture for deployments where the three tiers are installed on different computers. The Windows Communication Framework (WCF) is used as the communication protocol.
  • It improves security, since in the SQL database there is no need to create a SQL user with login name for each of the required NAV users. As a result, security is simpler, since there is no need for an improved security model to previous versions (client-server connection).

Great stuff! However, a side effect is that it is difficult to see which SQL Server connection (SPID) is assigned to a NAV session. This is a problem when trying to determine a Windows user to a table locking/blocking issue.
In the days of the classic client where you could see the active database sessions and blocking information along with the ability to terminate sessions if necessary are unfortunately history due to the new architecture.

Since customers keep asking us about the problem of identifying table locks in Dynamics NAV and if we didn't have a tool or solution for this problem, we developed a solution. Our first approach to a solution was our V8 NAV SQL Studio. However, the architecture of the software was suboptimal when installing Dynamics NAV service animals on different computers.

Monitor Microsoft Dynamics NAV server events:

For better understanding we would like to briefly explain the difference between Locking, Blocking and Deadlocking for all non SQL Server specialists. Locking: Locks are a mechanism used by Microsoft SQL Server to synchronize simultaneous access to the same data by multiple users.
Before a transaction retrieves a dependency for the current state of data, for example by reading or modifying the data, it must protect itself from the effects that may result if another transaction modifies the same data.

Blocking:

A BLOCKING occurs when two connections need access to the same data item at the same time and one connection is blocked because only one connection can have access at any given time.

This message then appears in NAV:

Microsoft Dynamics NAV
—————————

The operation could not be completed because a record in the '...' table was locked by another user. Perform the action again.
---------
OK
---------

Deadlocks:

DEADLOCKS occur when two tasks permanently block each other because each of the tasks maintains a lock on a resource needed by the other task.

Consequently, transaction A cannot be completed until transaction B is completed. However, transaction B is blocked by transaction A.

This message then appears in NAV:

Microsoft Dynamics NAV
—————————

The operation could not be completed because a record in the '...' table was locked by another user. Perform the action again.
---------
OK
---------

Locking is an integral part of Microsoft SQL Server to ensure parallelism and physical integrity of each transaction. Blocking is bad when a connection/transaction waits unnecessarily for a long time, and deadlocking is a phenomenon that should never occur.

V8 Search XE - Detect and fix deadlocks and blockages in Dynamics NAV Server.

V8 Search XE Ereignisverfolgung

Event tracing in V8 Search XE provides detailed information about what occurs on the Microsoft Dynamics NAV server when users are working with Microsoft Dynamics NAV and blocking or deadlocks occur. All data on specific Dynamics NAV trace events is captured in a V8 SQL Server database. This can help you identify and analyze issues or conditions affecting Dynamics NAV / SQL Server performance.

Identify table locks in Dynamics NAV. With V8 Search XE event tracking, you can dynamically monitor Microsoft Dynamics NAV servers without restarting the server or Microsoft Dynamics NAV clients.

You can use the V8 Search XE to track, for example, the following operations on Microsoft Dynamics NAV Server instances and the SQL Server:

  • Execution of SQL statements from Microsoft Dynamics NAV Server.
  • Execution of NAV C/AL functions.
  • Execution of Microsoft Dynamics NAV reports, queries and XMLports.
  • Process number, status, locks and commands (TSQL and C/AL or AL) executed by the active users.
  • Locked objects as well as the type of locks set up.
  • Full Dynamics NAV SQL trace of a locking process (SPID | DYNAMICS NAV USER).
  • Full Dynamics NAV SQL trace of active users by total waiting time (SPID | WAITTIME).

Which Dynamics BC / NAV Windows user locks the table?

V8 Search XE Tutorial Video



Read more…

We will be happy to personally answer any further questions you may have on this topic. Simply contact us via our contact form or by e-mail to info@dynamicsproject.com!

Your team from DynamicsProject.com

Back to Top