Skip to content

Latest commit

 

History

History
227 lines (141 loc) · 14.9 KB

CommandLineMOVES.md

File metadata and controls

227 lines (141 loc) · 14.9 KB

Running MOVES from the Windows command line

Introduction

While the easiest way to interact with MOVES is via its graphical user interface (GUI), it is sometimes useful to process MOVES RunSpecs in batches via the Windows command line. To this end, MOVES has a set of commands that can be accessed via the command line tool Ant. These commands can be further scripted using Batch files or PowerShell scripts. This document provides a brief introduction to the MOVES Ant commands and example batch scripts for running multiple MOVES RunSpecs in series. All of the examples presented here will use the Windows command shell (cmd.exe).

Set up the command environment

The first step to running MOVES from the command line is to set up the shell environment to point to all of the resources it needs to run MOVES. Open the Windows command prompt and navigate to the MOVES directory. By default, the path is C:\Users\Public\EPA\MOVES\MOVES3.0. The command to change directories is cd:

C:\> cd C:\Users\Public\EPA\MOVES\MOVES3.0

Then, enter the name setenv to run the setenv.bat script, which will set up your environment for you:

C:\Users\Public\EPA\MOVES\MOVES3.0> setenv

This will execute the setenv.bat script without displaying anything to the command window. The script tells the Windows command shell where to find the Java Runtime Environment (JRE) bundled with MOVES and where to find the Ant utility.

The rest of the instructions in this document assume that you have run these statements and are entering the subsequent commands in the same command prompt (except where noted).

View available MOVES Ant commands

After running setenv, MOVES Ant commands will be available in the Windows command prompt for the duration of the shell session. To test this you can type ant at the command prompt:

ant

This will display a list of the Ant commands that are defined in the MOVES build.xml file. Each command listed will include a brief description of what it does. There are commands included for recompiling MOVES, testing the MOVES code, and for running MOVES in a number of ways. This document will focus only on the basic commands needed for executing MOVES runs and using MOVES tools.

Running MOVES

A MOVES run can be launched from the command line. This method assumes that you have previously built a RunSpec (and any required input databases) using the GUI. Note that MOVES does not perform all of the same error checks before starting a run when launched from the command line as it does when launching from the GUI, so it is important to ensure that you do not have any errors in your RunSpec or input databases before starting a run using this method.

Launching MOVES master

The following command starts a MOVES run:

ant run -Drunspec="PathToRunSpec\RunSpec.mrs"

The RunSpec is specified using the -Drunspec= flag. The flag takes a relative path from the MOVES directory to the RunSpec to be run. By default, run will also start one worker to process bundles for the MOVES run as long as no other workers are detected. However, you may start additional MOVES workers using the command described below, which may speed up your MOVES run.

Launching MOVES workers

The following command launches additional workers to process bundles in a MOVES run.

Note: This should be executed from a separate command prompt (after running setenv to configure the new command prompt's environment), so that this can be running at the same time as the run command above.

ant 3workers -Dnoshutdown=1

3workers specifies that 3 workers should be started. Other options include 2workers and 1worker, which start 2 workers and only 1, respectively.

The option -Dnoshutdown=1 keeps the workers running if no bundles are received after a certain amount of time. This can be a helpful option when the bundles take a long time to be generated, or when running many back to back MOVES runs. However, note that worker processes started with this option can only be terminated by pressing Ctrl+C in the terminal window or by using the Windows Task Manager to kill the Java processes.

Note: If the 3workers (or the other variants) command is executed on the same computer before the run command is started, run will not launch another worker. However, if run is executed first, run will launch its own worker, and these workers will be in addition to that one.

If more than 3 workers are desired, there is an additional command that can be used:

ant manyworkers -Dmaxworkers=4

Here the option -Dmaxworkers=4 specifies the number of additional workers to start (in this case 4). Please note that having more than 3 workers is unlikely to improve performance.

Example script for executing MOVES batch runs

The above Ant commands can be used in batch scripts written to be run in the Windows command prompt. Batch scripts are text files with a .bat file extension that contain Windows shell commands that are run sequentially from the Windows command prompt. Two example batch scripts are shown below. The first is a simple example that demonstrates how to run multiple MOVES RunSpecs sequentially. The second shows how to start additional workers to potentially help speed up the MOVES run. Note that these scripts assume that all the RunSpecs show all green checks when loaded in the GUI (i.e., they are error-free) and that any input databases specified in the RunSpecs are available in MariaDB.

Basic MOVES batch script

Below is a simple MOVES batch script for running multiple RunSpecs sequentially.

:: Find current folder 
set RunSpecDir=%CD%

:: Set MOVES install location
set MOVESDir=C:\Users\Public\EPA\MOVES\MOVES3.0

:: Set up MOVES environment
cd /d %MOVESDir%
call setenv.bat

:: Run MOVES for each RunSpec 
:: (add a new line for each RunSpec)
call ant run -Drunspec="%RunSpecDir%\BatchTest1.mrs"
call ant run -Drunspec="%RunSpecDir%\BatchTest2.mrs"

:: Return to your RunSpec folder when everything is finished
cd /d %RunSpecDir%

In this batch script, the RunSpecs BatchTest1.mrs and Batchtest2.mrs are stored in the same folder as the script. The script finds its location, sets up the MOVES run environment, and executes the RunSpecs sequentially. After the last run is complete, the script returns the command prompt to the directory containing itself and the two RunSpecs.

MOVES batch script with extra workers

For computers with multiple CPU cores, MOVES performance can sometimes be improved by launching additional workers. Note that as multiple workers begin to compete for access to the MOVES database, their performance can decrease. Therefore, there are diminishing and potentially negative returns for launching too many workers on a given computer. In general, starting more than 3 workers is unlikely to improve performance. See Tips For Faster MOVES Runs for more information.

The script below is like the previous script, except that this script also launches three MOVES workers in a second command window.

:: Find current folder 
set RunSpecDir=%CD%

:: Set MOVES install location
set MOVESDir=C:\Users\Public\EPA\MOVES\MOVES3.0

:: Set up MOVES environment
cd /d %MOVESDir%
call setenv.bat

:: Launch 3 MOVES workers
start cmd /k ant 3workers -Dnoshutdown=1

:: Run MOVES for each RunSpec 
:: (add a new line for each RunSpec)
call ant run -Drunspec="%RunSpecDir%\BatchTest1.mrs"
call ant run -Drunspec="%RunSpecDir%\BatchTest2.mrs"

:: Return to your RunSpec folder when everything is finished
cd /d %RunSpecDir%

Once these MOVES runs finish, the workers will remain active until the user closes the command window that they are running in. Note that when the run command is executed, MOVES checks to see if any workers are running. It if doesn't find any, it will automatically start one worker to process its bundles. In the example above, the three workers are started before launching MOVES, so the run command will not start an additional worker.

Batch mode input database creation

MOVES includes an Ant command to create input databases, which can save time if you need to create many input databases at once. The following steps assume that you already have all of your input data files quality checked and ready to go.

  1. Using the MOVES GUI, create your base RunSpec.

  2. Using the Create Input Database Panel, create a corresponding database, and enter the Data Manager. Fully populate the database so that you get all green checks.

    Note: Do not close and re-enter the Data Manager during this step or before performing step 3. All tables must be populated in one session for step 3 to work.

  3. On the Tools tab of the Data Manager, click the Generate Importer XML File button, and give it a name. The file extension should be .xml.

  4. Make copies of the XML file for each database that you want to build on the command line. Open the file and make the following changes:

    • Within the <filters> tag, make any changes reflective of differences between the base RunSpec and the RunSpec that you will use with this database. For example, if the new database will be for a different calendar year, change the <year key="XXXX"> entry.
    • Name the database in the <databaseselection> tag. For example, <databaseselection servername="localhost" databasename="new_db" />
    • Update the <filename> and <section> tags for each input group to point to the corresponding data file and tab (if the file is .xlsx)

  5. For each input database to be created, run the following command (changing the path to the importer as appropriate):

ant dbimporter -Dimport="c:\mydbimporter.xml"

Converting input databases on the command line

MOVES includes an Ant command to convert input databases from previous versions of MOVES to the current version, which can save time if you need to do this for many databases. Please see the help file at database\ConversionScripts\InputDatabaseConverstionHelp.pdf for more information on the uses and limitations of this feature. Note that more work is needed after running this command before the converted database can be used with MOVES.

The following command can be used to convert a MOVES2014 (not a MOVES2014a or MOVES2014b) input database to the MOVES3 format:

ant convert2014_3 -Dinput=m2014_in -Doutput=m3_in

-Dinput is used to specify the old input database name, and -Doutput is used to specify the name for the new, converted database. convert2014_3 tells Ant to use the conversion scripts for going from MOVES2014 to MOVES3.

The following command can be used to convert a MOVES2014a or MOVES2014b input database to the MOVES3 format:

ant convert2014ab_3 Dinput=m2014b_in -Doutput=m3_in

-Dinput and -Doutput are the same as with the other conversion command, and convert2014ab_3 tells Ant to use the conversion scripts for going from a MOVES2014a or MOVES2014b input database to MOVES3 format.

Batch mode input database creation

MOVES includes an Ant command to create input databases, which can save time if you need to create many input databases at once. The following steps assume that you already have all of your input data files QA-ed and ready to go.

  1. Using the MOVES GUI, create your base RunSpec.

  2. Using the Create Input Database Panel, create a corresponding database, and enter the Data Manager. Fully populate the database so that you get all green checks. Note: do not close and re-enter the Data Manager during this step or before performing step 3. All tables must be populated in one session for step 3 to work.

  3. On the Tools tab of the Data Manager, click the Generate Importer XML File button, and give it a name. The file extension should be .xml.

  4. Make copies of the XML file for each database that you want to build on the command line. Open the file and make the following changes:

    • Within the <filters> tab, make any changes reflective of differences between the base RunSpec and the RunSpec that you will use with this database. For example, if the new database will be for a different calendar year, change the <year key="XXXX"> entry.
    • Name the database in the <databaseselection> tag. For example, <databaseselection servername="localhost" databasename="new_db" />
    • Update the <filename> and <section> tags for each input group to point to the corresponding data file and tab (if the file is .xlsx)

  5. For each input database to be created, run the following command (changing the path to the importer as appropriate):

C:\Users\Public\EPA\MOVES\MOVES3>ant -Dimport=c:\mydbimporter.xml dbimporter

Converting input databases on the command line

MOVES includes an Ant command to convert input databases from previous versions of MOVES to the current version, which can save time if you need to do this for many databases. Please see the help file at database\ConversionScripts\InputDatabaseConverstionHelp.pdf for more information on the uses and limitations of this feature. Note that more work is needed after running this command before the converted database can be used with MOVES.

The following command can be used to convert a MOVES2014 (not a MOVES2014a or MOVES2014b) input database to the MOVES3 format:

C:\Users\Public\EPA\MOVES\MOVES3>ant -Dinput=m2014_in -Doutput=m3_in convert2014_3

-Dinput is used to specify the old input database name, and -Doutput is used to specify the name for the new, converted database. convert2014_3 tells Ant to use the conversion scripts for going from MOVES2014 to MOVES3.

The following command can be used to convert a MOVES2014a or MOVES2014b input database to the MOVES3 format:

C:\Users\Public\EPA\MOVES\MOVES3>ant -Dinput=m2014b_in -Doutput=m3_in convert2014ab_3

-Dinput and -Doutput are the same as with the other conversion command, and convert2014ab_3 tells Ant to use the conversion scripts for going from a MOVES2014a or MOVES2014b input database to MOVES3 format.

Additional remarks

The above examples demonstrate a simple way of running MOVES from the Windows command prompt. The example scripts should be considered just examples. Different MOVES install configurations may change some of the details needed to run the scripts. Likewise, the Windows command prompt offers a rich scripting environment that can be used to expand these examples to do things such as:

  • Archive the moveslog.txt generated with each run (see MOVESMaster.bat for an example of this)

  • Loop through an arbitrary set of RunSpecs by searching the folder for files with .mrs extensions

  • Copy input databases stored elsewhere into the MariaDB data directory

  • Execute custom SQL post-processing scripts with MariaDB

Further details for the suggestions above are outside the scope of this overview. However, online you can find many good scripting tutorials and user guides for writing batch scripts in Windows.