Controlling Batch Operations


See About this document

See About the Batch Controller

See Setting activity times and triggers

See General Batch Controller parameters

See Connection parameters

See Time-to-run parameters

See Log-file parameter

See Timeout-limit parameters

See Example of parameters

See Handler identification

See Occurrence-driven execution

See Timed execution

See Metronomic execution

See Scheduled execution

See About SampleHandler

See Copying SampleHandler

See Customizing SampleHandler

See Configuring the Batch Controller

See Starting the new batch handler


 

About this document

This document explains how to configure the Batch Controller to launch batch handlers to check or process data for your Infranet ® system.

About the Batch Controller

The Infranet Batch Controller lets you specify when to run programs or scripts automatically, either at timed intervals or upon creation of certain files, such as log files. The Wireline Manager and other Infranet features routinely use the Batch Controller.

The Batch Controller configuration file ( Infranet_home /apps/batch_ controller/Infranet.properties ) has entries that tell the Batch Controller when to run the specified batch handlers. These programs or scripts can be triggered at specified times, or by specified kinds of occurrences, such as creation of a new log file. You can specify scheduled execution , starting the handler at fixed times of day only, or metronomic execution , starting the handler repeatedly at a fixed interval.

A batch handler can be any executable program or script that can be run from a command line. For more information about the requirements for writing batch handlers, see Writing Custom Batch Handlers .

Only one Batch Controller can run on a single computer, but it can control many batch handlers to launch various applications. Infranet installation includes a generic batch handler, called SampleHandler, which you can use with most applications. See About SampleHandler.

Setting activity times and triggers

Certain general parameters apply to all batch activity. Other parameters identify the batch handlers and regulate when those handlers are to be run.

General Batch Controller parameters

The Batch Controller's properties file specifies how to connect the Batch Controller to the Connection Manager (CM) , when to allow high batch traffic, how much logging to record, and how long to wait for handlers.

Connection parameters

Regardless of the type of triggering, you need to give the Batch Controller some parameters for connecting to a CM: a user ID, its password, the CM's port number, and the address of the CM's host computer. The Batch Controller logs on as the user you specify inthe infranet.connection parameter. If you want the logon to use both the user ID and password, set infranet.login.type = 1; if you want to not use the password, set infranet.login.type = 0.

In the batch.lock.socket.addr parameter, you must specify the socket number to lock on. If in doubt, check with your system administrator for the number of an unused socket that can be used exclusively by the Batch Controller. You needn't change the default value, 11971, unless some other application is using that socket number.

If you want to write a stack trace of any runtime exceptions in the log file, set infranet.log.logallebuf to true . To disable this feature, set it to false .

Time-to-run parameters

When your system's heaviest load typically occurs, you might want some handlers to run less often than they do in slacker times. The Batch Controller divides the day into a high-load period and a low-load period to let you balance the demand for system resources.

Specify the starting time of your system's busiest period in the batch.start.highload.time parameter. Specify the starting time of your system's slowest period in the batch.end.highload.time parameter. For each of these times, specify the hour, minute, and second in this format: hhmmss, using a 24-hour scale. For each handler, you can specify the maximum number of simultaneous actions during each of these two periods.

The end parameter must be greater than the start parameter. If you do specify start , it must be greater than end . Specifying the same value for both parameters causes an error.

In the batch.check.interval parameter, specify the time, in seconds, between checks for occurrences of the type specified in batch.timed.events or batch.random.events . Omitting batch.check.interval causes an error.

Log-file parameter

For logging, you can specify the level of information reported, the full path of the log file, and whether to print a stack trace of runtime exceptions. Set infranet.log.level to 0 if you want no logging, 1 for error messages only, 2 to also include warnings, or 3 to also include debugging messages. Set infranet.log.file to the path and file name for the Batch Controller log file.

If you omit infranet.log.file , Infranet uses default.pinlog in the current directory. Omitting infranet.log.level causes an error.

Timeout-limit parameters

You can also set timeout limits for handlers to start their objects and to complete their execution. Set batch.handler.start.wait to the number of seconds allowed for the handler to update its own object status from STARTING to STARTED , and set batch.handler.end.wait to the number of seconds allowed for updating from STARTED to some other state, such as COMPLETED . Writing Custom Batch Handlers describes all of the handler states.

Omitting either batch.handler.start.wait or batch.handler.end.wait causes an error.

Example of parameters

This example demonstrates the general parameters described above:


  infranet.connection pcp://root.0.0.0.1:mypass@myserver:11960/service/pcm_client
  infranet.login.type 1
  infranet.log.logallebuf true
  batch.lock.socket.addr 11971
  batch.start.highload.time 083000
  batch.end.highload.time 211500
  infranet.log.file /opt/portal/6.1/apps/batch_controller/batch.pinlog
  infranet.log.level 2
  batch.handler.start.wait 600
  batch.handler.end.wait 43200

 

In this example, the Batch Controller logs on to the CM host on myserver , port 11960, as user root.0.0.0.1 , using password mypass . High usage is expected between 8:30 a.m. and 9:15 p.m. Logging writes a normal level of information to file batch.pinlog and prints a stack trace if any program errors are found. Timeouts for updating object status are 10 minutes (600 seconds) for going to STARTED and 12 hours (43,200 seconds) for going to COMPLETED or some other state.

Handler identification

To identify each of the batch handlers:

  1. In the handler_id .name parameter, enter a brief descriptive label for the handler. This name can include spaces or any other characters. It can be of any length, but brief names are easier to read.
  2. In the handler_id .max.at.highload.time parameter, specify the highest number of instances of this batch handler permitted to run simultaneously during the time from batch.start.highload.time to batch.end.highload.time .
  3. In the handler_id .max.at.lowload.time parameter, specify the highest number of instances of this batch handler permitted to run simultaneously during the low-usage time--the time from batch.end.highload.time to batch.start.highload.time .
  4. In the handler_id .start.string parameter, specify the command line to start this batch handler.

Note When the Batch Controller issues that command, it appends
-p handler_poid -d failed_handler_poid to the command, as described in Writing Custom Batch Handlers . If you are not using custom batch handlers, you can ignore these options.

This example demonstrates the identification parameters described above:


  handler1.name Visa Handler #1
  . . .  
  handler1.max.at.lowload.time 4
  handler1.max.at.highload.time 2
  handler1.start.string /opt/portal/6.1/apps/visa-handler/visa.pl
     
  handler2.name Discover Handler #1
  . . .  
  handler2.max.at.lowload.time 5
  handler2.max.at.highload.time 3
  handler2.start.string /opt/portal/6.1/apps/discover-handler/discover -y 2000

 

In this example, the internal name Visa Handler #1 applies to the program started by the command string /opt/portal/6.1/apps/visa-handler/visa.pl , which runs a Perl script. The parameters shown here limit this program to one or two concurrent actions during the specified period of high expected load, or as many as four during the low-load period.

The other batch handler in this example, Discover Handler #1 , runs the application /opt/portal/6.1/apps/discover-handler/discover with the additional option -y 2000 .

Occurrence-driven execution

To trigger execution based on specified occurrences:

  1. In the batch.random.events parameter, specify the event identifiers. If you have two or more event identifiers, separate them with commas.
  2. In the event_id .name parameter, enter a brief descriptive label for the event. This name can include spaces or any other characters. It can be of any length, but brief names are easier to read.
  3. In the event_id .handlers parameter, specify the identifiers of one or more handlers that are to be triggered when the event occurs. You must specify at least one handler. If you have two or more handlers, separate them with commas.
  4. In the event_id .file.location parameter, specify the full path name of the directory you want to monitor for the arrival of new files matching the pattern in event_id.file.pattern.
  5. In the event_id .file.pattern parameter, specify the file-name pattern to look for.

Tip You can use an asterisk (*) to represent zero or more characters in the file name. No other wildcards (metacharacters) are supported.

Caution! You must specify batch.timed.events or batch.random.events or both. Specifying neither causes the Batch Controller to shut down just after it is started, because it is given nothing to do.

When the Batch Controller is started, it tests the file-name pattern against every file name in the specified directory and runs the batch handler for each file whose name matches the pattern. Then it monitors the files coming into that directory and runs the batch handler whenever it finds a match.

This example demonstrates the occurrence parameters described above:


  batch.random.events event1, event3
  . . .  
  event1.name Random Discover file arrival
  event1.handlers handler1
  event1.file.location /apps/discover/stagingDir
  event1.file.pattern  *.*.6011.*
  . . .  
  event3.name  Random Visa file arrival
  event3.handlers handler3
  event3.file.location /apps/visa/stagingDir
  event3.file.pattern *.dat
  . . .  
  handler1.name Discover UEL Handler
  handler1.max.at.lowload.time 6
  handler1.max.at.highload.time 3
  handler1.start.string /apps/discover -uel
  . . .  
  handler3.name Visa UEL Handler
  handler3.max.at.lowload.time 6
  handler3.max.at.highload.time 3
  handler3.start.string /apps/visa -uel

 

In this example, event1 is triggered when any file name in the /apps/discover/stagingDir directory matches the pattern *.*.6011.* . This match causes the Batch Controller to issue the command /apps/discover -uel , which runs handler1 , the Discover UEL Handler program. Based on the parameters shown here, the Batch Controller starts no more than six concurrent instances of this program during the specified period of low expected load, or three during the high-load period.

Timed execution

The Batch Controller provides two time-based scheduling options: See Metronomic execution. and See Scheduled execution..

Metronomic execution

To set up metronomic execution:

  1. In the batch.timed.events parameter, specify the event identifiers. If you have two or more event identifiers, separate them with commas.
  2. In the event_id .name parameter, enter a brief descriptive label for the event. This name can include spaces or any other characters. It can be of any length, but brief names are easier to read.
  3. In the event_id .handlers parameter, specify identifiers for one or more handlers that are to be triggered when the event occurs. If you have two or more handlers, separate them with commas.
  4. (Optional) In the event_id .start parameter, specify when you want the first execution to occur, in this format: hhmmss , using a 24-hour scale. If you omit this parameter, the first execution occurs immediately after the Batch Controller starts.
  5. In the event_id .interval parameter, specify the frequency, in seconds, for triggering the associated handler. Failing to specify the interval causes an error.
  6. (Optional) In the event_id .count parameter, specify how many times to execute this batch handler. If you do not specify this limit, batch handlers run repeatedly at the fixed interval, for as long as the Batch Controller is running.

Caution! You must specify batch.timed.events or batch.random.events or both. Specifying neither causes the Batch Controller to shut down just after it is started, because it is given nothing to do.

This example demonstrates the metronomic parameters described above:


  batch.timed.events event4
  . . .  
  event4.name Hourly tally
  event4.handlers handler4
  event4.start 000000
  event4.interval 3600
  event4.count 4

 

In this example, the occurrence specified as event4 is named Hourly tally . It runs handler4 for the first time at midnight ( 000000 ), then runs it again every hour (3,600 seconds) after that until it has run four times. If it cannot run at a scheduled time because previous executions are not finished, it runs again immediately as soon as possible. For example, consider this time line for event4 , above:

 
 
(Hourly tally starts)

In this example, the first run of handler4 continues for more than an hour, taking it past the time when the second run is supposed to begin. The second scheduled run cannot start at the one-hour interval, so it starts as soon as possible after that (1:20 a.m.). The third and fourth scheduled executions start at regular multiples of the interval, measured from the start.

If the overly long run continues past two scheduled starting times (occurrences), only one run is started on a delayed basis. For example, suppose the midnight run lasts until 2:25 a.m., as shown here:

 

In this case, the run scheduled for 2:00 begins immediately at 2:25. The run scheduled for 3:00 begins on time, as does the fourth and final run at 4:00.

Scheduled execution

To set up scheduled execution:

In the batch.timed.events parameter, specify the event identifiers. If you have two or more event identifiers, separate them with commas.

In the event_id .name parameter, enter a brief descriptive label for the event. This name can include spaces or any other characters. It can be of any length, but brief names are easier to read.

In the event_id .handlers parameter, specify identifiers for one or more handlers that are to be triggered when the event occurs. If you have two or more handlers, separate them with commas.

In the event_id .at parameter, specify each time when you want execution to occur, in this format: hhmmss , using a 24-hour scale.

This example demonstrates the schedule parameters described above:


  batch.timed.events event5
  . . .  
  event5.name Sporadic tally
  event5.handlers handler5
  event5.at 004500, 0022500, 045500
  . . .  
  handler5.name Sporadic Handler
  handler5.max.at.lowload.time 8
  handler5.max.at.highload.time 4
  handler5.start.string /apps/sporadic -uel

 

In this example, the program specified as event5 is named Sporadic tally . It runs only at 12:45 a.m., 2:25 a.m., and 4:55 a.m. If it cannot run at a scheduled time because previous executions are not finished, it runs again immediately as soon as possible.

About SampleHandler

Infranet installation includes a sample batch handler, called SampleHandler, which you can use with any batch application that processes data from files. The Batch Controller can call this batch handler whenever a specified directory receives a file whose name matches a specified pattern. The input and output files are then moved to specified directories.

SampleHandler is very similar to the mediation batch handler described in Configuring the mediation batch handler . A batch handler derived from SampleHandler is distributed with MPLS VPN (Multi-Protocol Label Switching Virtual Private Network) Manager. (See Using Infranet MPLS VPN Manager .) These are optional managers that your installation might not include.

To prepare SampleHandler for use, you copy the SampleHandler directory, modify the copied configuration files, modify the Batch Controller configuration file, then start or restart the Batch Controller.

If SampleHandler does not meet your needs, you can write your own batch handler, as described in Writing Custom Batch Handlers .

Copying SampleHandler

The directory Infranet_home /apps/sample_handler contains these files:

pin.conf --the batch handler's configuration file for Infranet-related parameters

sample_template.xml --used by the Universal Event Loader only

SampleHandler.bat (Windows) or SampleHandler.pl (UNIX)--the actual code of the batch handler

SampleHandler_config.values --the batch handler's configuration file for application-specific parameters

SampleReload.bat (Windows) or SampleReload.pl (UNIX)--used by the Universal Event Loader only

Copy the entire directory and name the copy appropriately. For example, if your new handler is to work with the Widget application, then you might name the new directory Infranet_home /apps/widget_handler .

In the new directory, you can rename the values file to include the application's name, such as WidgetHandler_config.values . If you do so, you must also edit the SampleHandler.bat (Windows) or SampleHandler.pl (UNIX) file to change SampleHandler_config.values to the new name.

Customizing SampleHandler

To configure the new batch handler for the desired application:

  1. Open the pin.conf file for the batch handler ( Infranet_home /apps/widget_ handler/pin.conf , for example).
  2. Edit the Infranet connection parameters. For information, see Reference Guide to Infranet Configuration and Properties Files .
  3. Save and close the batch handler's pin.conf file.
  4. Open the values file for the batch handler ( Infranet_home /apps/widget_ handler/SampleHandler_config.values , unless you have renamed the file).
  5. Ensure that the $HANDLER_DIR entry specifies the full path to the directory containing the batch application's log, input, output, and other files.
  6. Edit the $FILETYPE entry to specify the file-name pattern to look for.

    The Batch Controller will run the batch handler for each file that is in the specified directory and whose name matches this pattern.

Tip You can use an asterisk (*) to represent zero or more characters in the file name. No other wildcards (metacharacters) are supported.

  1. (Optional) If you want the batch handler's log file to be in a different directory than $HANDLER_DIR , edit the $LOGFILE entry to specify the full path to the desired directory.
  2. Ensure that the $pinUELDir entry specifies the full path to the directory containing the application to be run.
  3. Edit the $pinUEL entry to specify the name of the application to be run.
  4. (Optional) If you want the batch handler to get input files from a different directory than $HANDLER_DIR , edit the $STAGING entry to specify the full path to the desired directory.

The batch handler will move input files from the $STAGING directory to the $PROCESSING directory, where the application will read them.

  1. (Optional) If you want the application to get input files from a different directory than $pinUELDir , edit the $PROCESSINGentry to specify the full path to the desired directory. This must be the same directory that is specified as the application's input directory.

The batch handler will move input files from the $PROCESSING directory to the $ARCHIVE or $REJECT directory, depending on the application's exit code. Successfully processed files go into the $ARCHIVE directory, and files with problems to into the $REJECT directory.

  1. (Optional) If you want the application's processed files to go somewhere other than $HANDLER_DIR , edit the $ARCHIVE and $REJECT entries to specify the full paths to the desired directories.
  2. Save and close the batch handler's values file.
Configuring the Batch Controller

Having essentially created a new batch handler, you must identify it to the Batch Controller. Edit the Infranet.properties file of the Batch Controller, as described in Handler identification .

Starting the new batch handler

As with any batch handler, you start this one by starting or restarting the Batch Controller. The Batch Controller monitors the newly specified file location for the arrival of files and, when a file does show up, the Batch Controller starts the new batch handler. For more information, see See About the Batch Controller..

Before using the new batch handler for production, you should try it on a test system before using it in your production system.