Dynimize User Reference



Version 1.2








dyn·imize

verb
      to dynamically optimize the in-memory machine code of a running process.

modern English: from a combination of dynamic and optimize.




1. Overview

Dynimize is a user mode program that quickly optimizes the in-memory machine code of target processes based on profiling information it gathers at run-time, thereby improving performance for some workloads. Dynimize typically runs as a background system process and does not require the source code of target programs that it optimizes.



2. Installation & Quick Start

To install Dynimize, run the following commands in a Linux terminal.

wget https://dynimize.com/install -O install
wget https://dynimizecloud.com/install.sha256 -O install.sha256
sha256sum -c install.sha256; if [ $? -eq 0 ]; then sudo bash ./install; fi



Use your access token to start a subscription license for your host.

sudo dyni -license=start -token=<access token>



To optimize any CPU intensive process who's exe is listed in the [exeList] section of /etc/dyni.conf, run:

sudo dyni -start



The command dyni -status will show target processes progressing from the "profiling", to "dynimizing", and then "dynimized" states. A process has been fully optimized once in the "dynimized" state:

$ sudo dyni -status


Dynimize is running
mysqld, pid: 8375, dynimizing



$ sudo dyni -status


Dynimize is running
mysqld, pid: 8375, dynimized



Dynimize can be shutdown with the following command:

$ sudo dyni -stop


If Dynimize is shutdown, all dynimized processes will remain dynimized.


To "undynimize" all dynimized processes, run

sudo dyni -undo


or alternatively restart the dynimized process. For example, the following command can restart the mysqld process for MySQL:

service mysql restart



To cancel a subscription license for a given host, run the following command on that host:

sudo dyni -license=stop -token=<access token>



To uninstall Dynimize, run:

sudo /opt/dynimize/UNINSTALL.sh




3. System Requirements

    x86-64 Linux Kernel version 2.6.32 or later

    4GB virtual memory (swap + RAM) per target process being dynimized in parallel. This is because Dynimize requires 4 GB per target process in the "dynimizing" phase, releasing the memory after that.

    Above virtual memory requirement increased to 6GB (swap + RAM) per target process being dynimized when the target is a MySQL 8 mysqld process.


3.1 VM specific requirements

There are no specific requirements for running inside KVM, Xen or VMware virtual machines. Therefore Dynimize can be run in most cloud environments.


3.2 Docker specific requiremets

Dynimize must be run in the same container as the target process being optimized. When run, the dyni process requires the ability to use the ptrace(), process_vm_readv(), process_vm_writev(), and perf_event_open() system calls to profile and interface with the target process being optimized. This requires the dyni process to be run with the CAP_SYS_ADMIN and CAP_SYS_PTRACE capabilities. Docker as of the time of this writing (Docker version 18) disables these system calls and capabilities by default. The simplest way to enable them is by starting a docker container with the --privileged flag. For example, $run -it --privileged my_image. Alternatively, see Docker's documentation on seccomp profiles for specifically enabling just these system calls.



4. Command Line Options

Synopsys dyni [options]
General Options
• -help Print the help message.
• -start Start Dynimize as a background daemon process.
• -stop Stop any invocation of Dynimize that is currently running.
• -status Print the state or phase of Dynimize and its optimized processes. States will be listed in the following format: target process executable name, pid, state.
• -undo Stop any other invocation of Dynimize that is already running, remove all optimizations already performed by Dynimize on target processes, and then exit. Dynimize can then be started at a later time and used to dynimize the "undynimized" processes again under different workload conditions.
• -redo The same operation as -undo, except that this invocation of Dynimize will continue running, and may now proceed to automatically optimize the same processes again, along with other processes.
• -reoptimize Allow Dynimize to automatically redynimize a process if it detects that its workload has drastically changed. This option is disabled by default. Reoptimization can only be performed if the previous optimization was done by the same Dynimize process without Dynimize being restarted. See the -pid=<PID> option on how -reoptimize affects it.
• -version Print the Dynimize version.
• -about Display product information.
• -log=<pathname> Specify a full pathname for the output log.
• -fastCompile Reduce the amount of time required to dynimize a process. This may limit the final speedup of any processes dynimized this way.
• -lowOverhead Reduce peak profiling overhead for target processes. This may limit the final speedup of any processes dynimized this way.
• -exe=<filename> The current Dynimize command will only apply to processes run from this executable file name. Do not specify the full pathname.
• -pid=<PID> The current Dynimize command will only apply to the process specified by this PID. This will cause Dynimize to exit after dynimizing a process unless used in conjunction with -reoptimize
• -update Download and apply any Dynimize updates.
• -checkForUpdates Report if there are any Dynimize updates to apply.
• -skipPrompt Skip any prompts verifying the actions to be performed.
 
Licensing Options
• -token=<access token> Specify your account access token for use with any licensing actions.
• -license=start Start a subscription license for the current host.
• -license=stop Stop the subscription license for the current host.
• -license=fixed Create a permanent license for the current host. The account used will be billed for this license at the moment this command is issued, rather than at the end of the month like subscription licenses.
• -license=pull Download a copy of the license key into /opt/dynimize/license-key
• -info=this Display the local license information of the current host system.
• -info=all Display the license information of every host system associated with this account.
• -name=<license name> Specify a name for the license associated with this host.
• -licenseServer Run Dynimize as a license server. Requires an internet connection. See Licensing isolated hosts.
• -licensePort=<port> Include this option on the client and server side if you are running your own Dynimize license server, to specify the port number used by your Dynimize license server. If not specified then the default port used is 14701. See Licensing isolated hosts.
• -licenseHost=<host> Include this option on the client side if you are running your own Dynimize license server to specify the IP address or hostname of your Dynimize license server. If not specified then the default license host used is license.dynimize.com. See Licensing isolated hosts.

*Note that the output log will list all options enabled at Dynimize startup and can be used to verify which options are in effect.



5. Configuration File

On startup, Dynimize is configured based on the settings in /etc/dyni.conf. The following is a description of these settings. Many of these settings can be overridden by command line options. Note that starting a line with the # character will cause that line to be skipped.

All options follow the line in the .conf file containing the string [options]. If any of those following options are missing, their default settings are used. Note that the < | > characters should not actually be placed in the log file:

Option Description
reoptimize Same as the command line option ‑reoptimize
log=<pathname> Same as the command line option -log
maxLogSize=<size> The max total size of both log files at <path> and <path>.old combined. Size is in bytes unless specified by MB, M, GB, G, KB, K. The defaults is 1 MB.
fastCompile Same as the command line option ‑fastCompile
lowOverhead Same as the command line option ‑lowOverhead
initdService If present, Dynimize will be enabled as service in /etc/init.d and launched on system startup.
autoUpdate If present, Dynimize will automatically check for Dynimize updates and apply them.
skipPrompt Same as the command line option ‑skipPrompt
token=<access token> Same as the command line option ‑token=<access token>
autoRenew If expired, automatically update the license key in /opt/dynimize/license-key.
licenseServer Same as the command line option ‑licenseServer
licensePort=<port> Same as the command line option ‑licensePort=<port>
licenseHost=<host> Same as the command line option ‑licenseHost=<host>


The lines following [exeList] in dyni.conf denote the whitelist of possible optimization targets. These are the names of the executables that are used to launch the processes that can be targeted by Dynimize, one name per line. So for example, the MySQL server process would be denoted as mysqld. A process must have been started by an executable on this list to be dynimized. Note that if a symbolic or hard link is used to call an executable, the real executable file name must be specified on the [exeList].

The lines following [users] list the possible user names of the owners of processes that can be targeted by Dynimize, one name per line. If present, a process must match both the [exeList] and [users] list in order to be dynimized. Default: no user name requirement.

The output log specified by -log=<pathname> records all errors and Dynimize status updates along with their timestamps. It will also list all options enabled at Dynimize startup which can be used to verify the options that are in effect. On installation the log path is set to /var/log/dyni.log in /etc/dyni.conf. Once the log file reaches half the maxLogSize, it is moved to the same path name appended with .old, and a new log is started as the original log file name. Logging is disabled if the log option is removed from dyni.conf.



6. Dynimize Usage Examples

Start Dynimize as a background daemon process, dynimize any processes encountered, and then continue to run and target new processes that are CPU intensive and on the [exeList] in /etc/dyni.conf.

$ sudo dyni -start



Dynimize a process with pid 9618 and exit once dynimized:

$ sudo dyni -start -pid=9618



Launch Dynimize as a foreground process, only dynimizing process 9618 and then exit. Note that the -start option is excluded here so that Dynimize is not launched in the background, which can be useful for running Dynimize from shell scripts that wait for Dynimize to finish before performing the next task:

$ sudo dyni -pid=9618



Launch Dynimize, only dynimizing process 9618, however in this case Dynimize will continue to run after the process has entered the "dynimized" state so that it can reoptimize the process if its workload drastically changes:

$ sudo dyni -start -pid=9618 -reoptimize



The following example will stop any other running instances of Dynimize, remove any optimizations performed by previous Dynimize invocations, and then continue running. Dynimize may now proceed to optimize these processes again, along with any other processes. This could be useful if you think that your workload has drastically changed and want it reoptimized using current profiling information:

$ sudo dyni -start -redo



Only dynimize processes run from the myprog executable:

$ sudo dyni -start -exe=myprog



Dynimize processes and shorten the amount of time spent in the dynimizing phase:

$ sudo dyni -start -fastCompile



Start Dynimize with a new log path directory /tmp/log:

$ sudo dyni -start -log=/tmp/log




7. Licensing & Billing

One of the primary purposes of Dynimize is to reduce the financial costs of CPU resources consumed by target workloads for a given level of performance. Therefore to help users reduce billing costs in the cloud, elastic, on-demand subscriptions can be generated on the command line to license Dynimize with per second billing granularity, thereby matching the billing granularity of CPU resources in many public clouds and allowing for easy licensing automation.

Use your access token to start a subscription license for your host.

sudo dyni -license=start -token=<access token>


To cancel a host's subscription license from the command line, Dynimize must be shut down and there cannot be any dynimized processes running on that host.

To stop running instances of Dynimize and "undynimize" all dynimized processes, run

sudo dyni -undo


or you can simply stop Dynimize and restart any dynimized processes to "naturally" undynimize them. For example, the following commands can restart the mysqld process for MySQL after stopping Dynimize.

sudo dyni -stop


service mysql restart


To then cancel a subscription license for a given host, run the following command on that host.

sudo dyni -license=stop -token=<access token>


By default, time is accounted for using per second granularity, but only when the license is later correctly cancelled from the command line. See Billing time granularity for exceptions.

Time accounting for any licenses used only begins 7 days after your account was activated. This initial period is your free trial, after which each license is then priced at $1 per CPU core per 30 days, prorated per second of usage when cancelled from the command line on the licensed host.

For example, if you only have 1 hour and 48 minutes of use over the course of a month on a 4 core host, you will only be invoiced $0.01. Your account will be billed at the end of each month for subscription license time used during that month.


7.1 Fixed licenses

A fixed license is only billed once per host, is priced at $24 per CPU core, and will support a single host indefinitely. You can start a fixed license with the following command.

sudo dyni -license=fixed -token=<access token>


A fixed license can also be created on a host anytime after a subscription license was started. In that case, the subscription license is implicitly stopped and no subscription time will be billed past that point for that host. A fixed license is then created at the standard price.


7.2 Viewing licenses

To view all the active licenses associated with your account, run

sudo dyni -info=all -token=<access token>


To only view the license on the current host, run

sudo dyni -info=this -token=<access token>



7.3 Storing your access token

Your access token in the option -token=<access token> acts as the credentials for accessing the Dynimize licensing system. To prevent it from being stored in your bash history, you may want to change your access token and then use that new value in the configuration file /etc/dyni.conf by placing it in the token=<access token> field. You can then omit -token=<access token> from the command line. Note that any use of -token=<access token> on the command line will automatically update the value in /etc/dyni.conf.


7.4 License key expiration

Licensing commands will store the current license key in /opt/dynimize/license-key. While fixed license keys never expire, subscription license keys will report that Dynimize is expired 30 days after license key creation, and prevent Dynimize from starting or performing further optimizations (if already running) after 60 days from license key creation. If the flag autoRenew is set in /etc/dyni.conf and the server has internet access or network access to a self hosted license server, Dynimize will automatically update the license key every 30 days. This provides an additional 30 day window to address any issues encountered renewing the license key.


7.5 Anonymous bookkeeping

The licensing system creates a relatively unique identifier for each licensed host. This identifier combines system values such as the MAC address and the number of CPU cores. This information is combined into a 32-bit identifier that contains a fraction of the total bits of the original system values, in such a way that those original values cannot be extracted from this unique identifier. This allows Dynimize to associate a license with a specific host without transmitting any of the real system values of that host, maintaining a degree of anonymity and privacy. This identifier is then used to allow the licensing system to ignore redundant licensing commands. For example, starting a subscription license on a host that already has a fixed or subscription license will have no effect.

Note that all communication between a host and a license server is encrypted through TLS. This communication only happens when a licensing command is issued by the user, or a subscription license key is automatically updated once expired, which is once every 30 days only if autoRenew is enabled and a subscription license is in use. Dynimize does not require a network connection to the licensing system outside of those times.


7.6 Billing time granularity

Dynimize time accounting is performed with per second billing granularity only when a subscription license is stopped from the command line. However, if /opt/dynimize/dyni-node is modified on the host at any point after license key creation, per second billing granularity will not be possible for the next licensing command and the host may continue to accrue billable time for an additional 30 days after the subscription license is stopped from the command line. Additionally, if the MAC address or number of CPU cores on the licensed host has changed, Dynimize licensing commands will fail to recognize the host system and the subscription license will need to be stopped from the web portal, where per second billing granularity cannot be used. In this case, Dynimize will continue to accrue billable time for an additional 30 days after the subscription license is stopped from the web portal.

IMPORTANT: For the above reasons, it is important to stop a subscription license from the command line of any cloud instance before terminating that instance. This is because any new instance will have a different MAC address, requiring the original instance license to be stopped from the web portal and thereby forfeiting per second billing granularity.


7.7 Licensing isolated hosts

Some security policies do not allow specific servers access to the internet, but allow them to reach internal servers that do have internet access. In this case you can run Dynimize as a licensing server on your own system that does have internet access and is reachable by these isolated servers you wish to license.

To run Dynimize as a licensing server background process, run the following command on the host with internet access.

sudo dyni -start -licenseServer


Alternatively, you can set licenseServer in /etc/dyni.conf

You can then run any licensing command on the isolated servers that can only reach the above host using the -licenseHost=<IP|host> flag. For example, to start a subscription license on an internet isolated host assuming the above self-hosted license server has an IP address of 10.30.1.72, run the following command.

sudo dyni -license=start -licenseHost=10.30.1.72 -token=<access token>


You can also set licenseHost=<IP|host> in /etc/dyni.conf on the isolated servers. Note that the hostname can be used in place of the IP address in the above example.

By default, all licensing commands use port 14701. To change this in the above example, on both systems use the command line option -licensePort=<port number> or set licensePort=<port number> in /etc/dyni.conf.




8. Workload Requirements

To obtain benefit from the current version of Dynimize, all of the following workload conditions must be met:

A small number of CPU intensive processes
On a given OS host where the workload is running, the workload must be comprised of one or a few CPU intensive processes. Optimizing a large number of processes at once is not recommended.

Long running programs
The processes being optimized have long lifetimes, and their workloads are long running in order to amortize the warmup time associated with optimization.

Linux x86-64
Optimized processes must be 64-bit, derived from x86-64 executables and shared libraries, which must comply with the x86-64 ABI and ELF-64 formats. Most statically compiled applications on Linux meet this requirement.

Dynamically Linked
Target processes must be dynamically linked to its shared libraries. Statically linked processes are not yet supported. Most Linux programs are dynamically linked.

No self modifying code
The target application must not be running its own Just-In-Time compiler such as those found in Java virtual machines. This therefore excludes Java Applications.

Front-end CPU stalls
The workload wastes a lot of time in CPU instruction cache misses, instruction TLB misses, and to a lesser extent branch mispredictions.

User mode execution
Much of that wasted time is spent in user mode execution (as opposed to kernel mode), as Dynimize only optimizes user mode machine code.

Because of these requirements, Dynimize takes a whitelist approach when determining if programs are allowed to be optimized, with MySQL and its variants being the currently supported optimization targets on that list. Other programs are not currently supported, and while many can be used with Dynimize, they should be very thoroughly tested with Dynimize before being deployed in a production environment.

Future versions of Dynimize may eliminate many of these workload requirements, broadening the variety of applicable scenarios as well as further increasing the performance delivered in previously beneficial cases.



9. Miscellaneous Notes

Sequential CPU speedup
Dynimize usage may result in the speedup of the target application and/or a reduction in CPU resources consumed by that application. Speedup typically occurs when sequential CPU performance is a bottleneck. When sequential CPU performance is not a bottleneck, a reduction in CPU resources used by the optimized process is experienced which can be observed by an increase in CPU idling. All improvements are generally limited to time spent in user mode execution.

Dynimize performs work in response to CPU usage
When a process whose executable is listed in the exeList consumes large amounts of CPU resources, Dynimize will automatically begin to optimize the in-memory machine code of that process. Dynimize only performs work in response to CPU resources consumed by a target process. The more CPU resources consumed by a target process, the more intensely Dynimize will work to dynimize it and the more quickly the process will become dynimized. A target process that consumes little CPU resources may never be targeted by Dynimize, or will take a long time to progress to the dynimized state.

Dynimize can respond to changes in workload
Once all target applications are sufficiently optimized, Dynimize will enter idle mode. If Dynimize was originally started with the reoptimize option, it may then respond at a later time to a significant change in the CPU workload of an already dynimized process by redynimizing it. It will also dynimize any new CPU intensive undynimized process that was started from an executable on the exeList.

Zero downtime
Target applications do NOT need to be restarted in order to be dynimized. Once Dynimize is started it will automatically detect and begin dynimizing them immediately.

One Dynimize instance per OS host
Dynimize is not designed to be run as multiple instances on the same host OS. If a second instance is launched in parallel, it will detect an already running instance of Dynimize and exit.

Single threaded
Dynimize is currently single threaded and will only consume the resources of at most one CPU core, even when optimizing multiple target processes in parallel.

No handoffs between Dynimize instances
A new Dynimize process will not dynimize a target process that has already been dynimized by a previous Dynimize process, unless that target process has been undynimized using the -undo or -redo flags.

No stale shared libraries
Dynimize will not dynimize a running process if its shared libraries on disk have changed after they were loaded into that running process. dyni -status will report "stale shared libraries" for that target process and it must be restarted before Dynimize can target it.

Dynimized processes stay dynimized
Once Dynimize is shut down, any dynimized processes will continue to run in the dynimized state. However if a dynimized process is then restarted, any previous optimizations will have disappeared and Dynimize will need to be started again to target the application.



10. Example Life Cycle of Dynimize and a Target Application

  1. Dynimize is started as a background process and begins in idle mode, monitoring system processes. This state consumes virtually no CPU resources:

       $ dyni -start


  2. A new target application process such as mysqld is then launched and becomes CPU intensive, or an already running target application becomes CPU intensive.

  3. Dynimize detects this and begins to dynimize the target application. Incremental, atomic updates to the target application's machine code are made. If the target application remains CPU intensive then this optimization typically takes around 60 seconds to complete:

       $ dyni -status


      Dynimize is running
    mysqld, pid: 8375, dynimizing


  4. Dynimize has completed its current batch of optimization work and enters idle mode:

       $ dyni -status


      Dynimize is running
    mysqld, pid: 8375, dynimized


  5. A new target application is launched and Dynimize returns to step 3. Additionally, if the workload of the previously dynimized process has drastically changed and Dynimize was started with the reoptimize option, it also returns to step 3. Although Dynimize is single threaded, it can apply these steps to multiple target processes at the same time.


11. Dynimize Overheads

Dynimize introduces both CPU and memory overheads when ramping up performance during the dynimizing phase, with plans to reduce these overheads in future releases. The following section addresses these overheads.


11.1 Dynimize CPU overhead

CPU overhead exists during the warm-up phase when a process is being dynimized. There are two components of CPU overhead during this phase. The most obvious is the amount of CPU cycles that the dyni process is actually consuming. While at first spiking to 100% utilization of a single core for less than a second, dyni typically fluctuates at around 20% CPU utilization (of a single core) for the remainder of the warm-up phase. Because it is quite rare to be fully utilizing all hardware threads on a large multi-core system, the single threaded dyni process is unlikely to result in significant overheads in this regards. The second CPU performance overhead that takes place is that of application profiling, and although brief, it is typically far more drastic. Both these overheads are offset by the gradual machine code optimizations that take effect, and are completely eliminated once the process reaches the dynimized phase. Therefore an initial warmup period lasting anywhere from 30 seconds to several minutes should be set aside for workloads using Dynimize.

11.2 Dynimize Memory overhead

Memory overhead also exists, where the dyni process typically requires around 4 GB of virtual memory for each target process being dynimized, while it is dynimizing them. Most of this virtual memory consumption is used for book keeping purposes when dynimizing, and so the dynamic range of memory accesses is quite limited and in our experience does little to trigger additional page faults in memory constrained workloads. That said, at least 4GB total of free swap space + RAM is required for each dynimized process during the dynimizing phase. Once a process is dynimized, most of that memory is freed. The dyni process then typically consumes 50-150 MB of virtual memory per dynimized process. This is used for book keeping purposes in order to react to drastic workload changes in dynimized processes. This memory also undergoes a very small range of dynamic accesses, and should therefore have negligible impact on system paging.

Note that when using -reoptimize, if a drastic workload change occurs for a specific process then that target process may be redynimized and will return to the dynimizing phase. In this case Dynimize will briefly incur the 4 GB virtual memory overhead per target process again.

An additional use of memory is the 35 MB code cache used by the just-in-time compiler that is loaded into the target process at the start of the first dynimizing phase. That being said, the actual access patterns of machine code instruction memory with the addition of the code cache are more constrained than that of the original undynimized process, and so the resident memory pages used there should be less in most cases. Note that the flags -undo and -redo will leave behind this 35 MB code cache in a target process, and any subsequent reoptimization of a target process by Dynimize will then create a new code cache when that same process first enters the dynimizing state again. On the other hand, when the -reoptimize flag is used and dyni detects a drastic workload change in a process already dynimized by the current dyni invocation and subsequently reoptimizes it, Dynimize will continue to use the same code cache initially setup by the current dyni invocation.

Overall, because of the limited range of dynamic memory accesses, these memory overheads should not affect performance in memory constrained environments. Therefore more RAM is typically not required. However one should make sure that the system swap space can accommodate the increased virtual memory used during the dynimizing phase in memory constrained environments.



12. Using Dynimize on Applications Other than MySQL

Many have asked why MySQL is the main target for the initial release of Dynimize. Database workloads can be IO bound and may not always benefit from improved CPU performance to the same extent as other workloads. Extreme levels of instruction cache misses are common in many other CPU intensive enterprise software workloads, which leaves many places where the current crop of optimizations in Dynimize could be put to productive use in the future. However a lot of these other examples such as Redis or Nginx spend most of this CPU time in system mode execution (in the Linux kernel). The machine code underlying this system mode execution cannot be optimized by Dynimize in the current release and therefore these applications benefit to a lesser extent than MySQL. For example, at most around a 10% speedup with Nginx in a CPU bound setup has been observed. Other programs such as Apache HTTP Server are often configured in a multiprocess configuration where there may be too many processes for Dynimize to handle effectively, or where the processes have short lifetimes. For these reasons, little time has been spent investigating these other use cases up to now.

Future versions of Dynimize will be able to optimize workloads with these characteristics as well. However at the moment, MySQL and its variants have been the main focus, with other single process, multithreaded relational databases on Linux such as MongoDB being likely candidates to benefit from this current release. Stay tuned for expanded scope in future releases.

If you find Dynimize useful in other workloads outside of MySQL, we'd love to hear about it. Let us know at info@dynimize.com.



Questions & Feedback

We love answering questions and your feedback is extremely valuable! Please use the discussion form below for anything related to this documentation.














COPYRIGHT © DYNIMIZE INC.