Unattended Testing Methods

From Bots-United Wiki

Jump to: navigation, search


Unattended testing methods covers the various ways that you can let the bots fight the waypointed maps for you while reporting back the results for later review. You can also spectate and watch them fight, but the goal is to not have to be there all the time. This topic is written for FritzBot ET testing but includes many methods that can be used by other bots.

Contents

[hide]

Components of a Test Plan

Image:Test-map.png

Inputs And Setup

Bots file

If the map has a bot file, review it and count the number of axis bots and the number of allied bots. They should be the same with almost identical makeup. There likely will be one instance of the Random bot name appearing in the mapname.bot file. This and any other should be commented out by typing 2 slashes // immediately to the left of the word random on the same line. If no botfile exists then it is simple to make one in a text editor like notepad. You can also find a bot-file creator in Hobbits waypointing tool. Note for best effect the bots should be added to the 2 teams like how you chose team mates when playing pick-up baseball (or whatever) as a Kid. That is add them one at a time alternating between the teams each time. Better still use the 1-2-2-..-2-1 team ordering for the least statistical bias. (In the event bots are still joining when the warmup ends a simple alternating order favors the team whose bot was first to join. I.e. interrupt the process and 50% of the time both teams have the same number and 50% of the time the first team has a single bot advantage).

Bots XP

FILL IN LATER, for now from the main page...

bot_skill <1-3>

Used to set the bots navigation and awareness setting. 1 is lowest skill, 3 is highest skill

bot_aimskill <1-4> default is...

Used to set the bots aiming skill. 1 is lowest level, 4 is highest level

If you want to add a variety of skill levels for the bots, you can specify ai skill and aimskill with the addbot command. These settings will override the server cvars for each individual bot as it connects: /addbot <botname> <aiskill> <aimskill>:

addbot fritz 1 1 //the lowest skill available
addbot dolphin 3 4 //the highest skill available

Test at Warp Speed

from Denny's tutorial

/cg_thirdperson 1
/cg_thirdpersonrange 150
/timescale 2 

(a timescale value of 1 turns the timescale back to normal, the timescale increases the speed of the game

These commands will work when testing with /devmap <mapname>, but the timescale change is considered a cheat at client level so it will not work with /map or /campaign.

Preconditioning The Map State

If doing a single pass test of a map, you can precondition the map state by pausing the bots ( /bot_pause 1) while you run around building and blowing up things etc. This manual method has to be identically set up for each test pass so it is not recommended for more than a couple of passes.

Modifying the map state in the script file

Likewise if you are very comfortable with map script files you can create a special mapname.script file that starts the game in an initial state with various objectives and things completed. You can move the game along even with the spawnpoints at a different configuration to simulate a later point in the game. This method can be used in self-repeating tests like campaigns or multiple single-objective testing since they reload the script file at each map load.

Since each <mapname>.script is different it is too complex to provide much in way of example here.

Map Config File

The mapname.cfg file in FritzBot can be used to execute console commands at the beginning of the map. Useful console commands to include here are;

g_axiswins - a 10-bit-packed variable representing the maps/matches in the current campaign that axis has won.

g_alliedwins - ditto but for the Allies.

currentTime - prints the local Human time to the client console

If you have added user cvars in the script file to count events (like team wins, or objective completed) then added the cvar name to the mapname.cfg to log their value at the beginning of play.

Campaign Test File

Now here is one way to run a 9 map cycle and prevent the campaign auto-restarting (and reseting the cvars)

{
	name            "mapname_9stop"
       shortname        "mapname_9stop" 
	description	"mapname only 9 cycles then stop:  For testing the team balance on mapname waypointing"
	maps		"mapname;mapname;mapname;mapname;mapname;mapname;mapname;mapname;mapname;nomap" 
	mapTC		374 374
	type		"wolfmp"
}

This campaign file is described in detail under the topic #Balance_Tests_on_a_9_map-cycle_campaign. In brief the campaign runs mapname until it gets to nomap where it halts the campaign since nomap does not exist.

Campaign or Map Cycles

To learn more about map or campaign cycles a good reference is Map rotations & Campaigns

In brief a map rotation is started from an exec command at the end of the server config file. The exec command runs another config file that you made where a number of cvar vstrings (variable strings) are defined. A cvar vstring is executed by the special command vstr, and may contain a number of commands separated by semi-colons. The key to making the rotation is that each cvar vstring invokes the next cvar vstring and that it uses the nextmap or nextcampaign cvar to do it.

so the form of the cvar string in the map rotation is

set <vstringname> "g_gametype <GT>; map <mapname>; set nextmap vstr <nextvstringname>" 

where <vstringname> is the cvar vstring you want to use for this part of the rotation, and <nextvstringname> is the cvar vstring that runs the next map. <GT> is the value for the gametype (2=singleObj, 3=stopwatch). <mapname> is the name of the map to run. (Same filename as the bsp file in the .pk3 or the .nav etc.)

Or in a campaign cycle

set <vstringname> "campaign <campaignname> ; set nextcampaign vstr <nextvstringname>"

where <campaignname> is the shortname of the campaign that will be run by the vstring <vstringname>.

Now the cycle can be started by /vstr <vstringname> in the console or by including a set com_watchdog_cmd command in the rotation config file to invoke the same command whenever no map is running.

Starting The Test Run

Clear the logs

Since you will likely want to refer to any generated logs just for the timespan of the test, you should clear out any existing logged messages before starting Enemy Territory (ET). This is done simply by deleting or re-naming the old log files. The Log files are normally named etconsole.log and etserver.log and reside in the directory of the modification being run (in \fritzbot\ for the FritzBot mod, and in \etmain\ for straight ET).

If you have never turned on continuous logging for ET you can do so now. See #Logging_To_Disk_(Log_Files).

If you are using a script or bind to generate condumps then delete, move or rename any of files that the script or bind command will use just in case to avoid any possible confusion.

Getting the bots to join

In FritzBot, to start the test cycle you will have to join the map to get the bots in the <mapname>.bots file to join in. Note that if you are presetting the map state that you can also kick all bots, do your preset thing then switch to spectate and FritzBot will re-add the bots-file bots that you kicked.

Outputs

Condumps

Manual Method:

/condump some_filename

Where some_filename is the filename and filetype of the file to be written to in the

\fritzbot directory.

This will dump the console messages in the buffer for this machine to the specified file.

However the buffer is limited to 30KB normally which may be as little as 5-15 minutes of play.

Logging To Disk (Log Files)

Using continuous log files...

Copy the script inside the box (below) into notepad and save it to \etmain with the filename

logit.cfg.

// FILENAME: etmain\LogIt.cfg
// USAGE: /exec LogIt.cfg
// OUTPUTS: \fritzbot\etconcole.log \fritzbot\etserver.log
//         for other bots substitute the mod name for fritzbot
// DESCRIPTION: How to log server and console messages
//
seta cg_drawNotifyText "1" // 0 or 1, Client Side, etconsole.log
seta cg_textNotify "1"     // 0 or 1, Client Side, etconsole.log
seta g_logSync "2" // Server log - 0=no log, 1=buffered, 2=continuous,  3=append
seta g_log "etserver.log"  // Server log filename 
//                  (use Log ? for setting server/client log filename?)
seta logfile "2" // Server and Console logging - 0=disable? 1=enable, 2=enable_and_sync

To run this config open the console(~ or `) and type

\exec LogIt.cfg

This changes the default for the option settings (above) for the current mod. You will notice that the console will become more verbose as the previously un-viewed server messages are included.

Saving and running the .cfg in the box below should return you to normal and kill the verbosity.

// FILENAME: etmain\LogOff.cfg
// USAGE: /exec LogOff.cfg
// OUTPUTS: \fritzbot\etconcole.log \fritzbot\etserver.log
//         for other bots substitute the mod name for fritzbot
// DESCRIPTION: How tp turn to default log setting
//
seta cg_drawNotifyText "0" // 0 or 1, Client Side, etconsole.log
seta cg_textNotify "0"     // 0 or 1, Client Side, etconsole.log
seta g_logSync "0" // Server log - 0=no log, 1=buffered, 2=continuous,  3=append
//seta g_log "etserver.log"  // Server log filename 
//                  (use Log ? for setting server/client log filename?)
seta logfile "0" // Server and Console logging - 0=disable? 1=enable, 2=enable_and_sync

Scripting Debug Messages

The stats generated in the console at the end of the map are your player and the other players' stats. These do not directly say which team won. Indeed each map has a different way of expressing winning objectives (if at all) in the console. What we need is a predictable message that we can parse. So simply we can add the message(s) we need to the script file, and then search the output file for that message(s).

There are generally 2 ways of putting a message into the console (and console.log). First there are the debug print commands. Second are the game announcements. The latter announcements are also shown in the game so are easier to verify as working.

The commands in the script file you need to search for are; wm_setwinner [-1, 0, 1] and... wm_endround The command "wm_endround" ends the game play sending the players into the post game stats menus and end of game effects (explosions, invading tanks etc.). This command may occur in one or more places in the map. Our end of game announcements should immediately precede this wm_endround command. Now the wm_setwinner command informs ET who won so it can be displayed on the world map and in the game end stats display. The command should occur once in the game_manager { spawn {}} event to define if there is a default winner (i.r the defender). The value -1 mean no winner (example dm_hillb2), 0 axis wins in the default case (e.g. battery, goldrush, radar, fueldump...), and 1 means allies win by default (e.g. mml_church_et_v1, trenches05 etc...). So here are the suggested forms of the debug message/announcements;

wm_announce "[WINNER0] FIGHT!" //place this in game_manager's spawn event.
wm_announce "[WINNER2] Allies win" //add before wm_endround when the allies win
wm_announce "[WINNER1] Axis wins" //add before wm_endround when the axis win

Scripting Counters

When there is a particular event that you need to know if and how many times it occurs, instead of reading and counting occurrences in the log file, you can script counters in the map script file. There are 3 basic types of counting variables in a script file. The 'accum n' where n-0..9, is a local script block variable that can be set or incremented inside its script block and can be tested and printed to the console inside the same script block. There is also a single set of 8 (0-7) usable globalaccums that can be set, incremented, tested or printed anywhere inside the map script file. Finally you can use user cvars (including those you define) as increment-only counters. These cvars can be tested and incremented anywhere in the map script file. (The cvar set command also exists but does work because the original ET source code in G_ScriptAction_Cvar does not call trap_Cvar_Set to update the cvar.) The difference between a globalaccum and a cvar, is the cvar retains its value between matches and can be set or output from within cfg files.

An example of a use of a scripted cvar in unattended testing might be counting wins in a single objective or stopwatch match that is allowed to restart itself. To do this you will need 2 cvars that are manually reset before first running the map (e.g from the console). The placement of the cvar increments will depend on the script file, ideally there will be a checkgame script block that can be used to increment the cvar for one of the winning teams. Many script files have a default winner for timelimit wins, and in that case it may be necessary to increment a total matches cvar counter near the beginning of the map rather than have each team have its own counter.

For example: defining 2 cvars; totalmatches, and axismatches in a map where allies win at timelimit hit.

game_manager
{
	spawn
	{
		...

		abortifwarmup  //prevent the double count up of the balance cvar  
		              //due to the spawn function being called during warmup
		cvar totalmatches inc 1  //NOTE this will now be equal to the number of matches completed 
					//PLUS the current one in progress TomTom
	}
...
	trigger checkgame
	{ 
		accum 2 abort_if_not_equal 0 //test if axis has winning condition
		
		wm_setwinner 0 //tells the wm_endround manager to award the match to team 0 -axis

		cvar axismatches inc 1  //True count of number of axis wins
	
		wm_endround		//End of the Game
	}
}   //end of game_manager

Note that the counter totalmatches would be double the actual value unless we prevent it being incremented during the warmup. Also note that if you disconnect ET in the middle of the match the totalmatches will be 1 greater than actual completed matches. Now to output these cvars regularly to the console just add them to the map.cfg like the g_axiswins g_alliedwins in the topic #Map_Config_File above and the #Balance_Tests_on_a_9_map-cycle_campaign mapname.cfg example below.

Parsing Log Files And Condumps

Any text editor can be used to search for individual instances of special messages such as errors and warnings. If running a later version of Windows (e.g. XP) these manual searches can be done in notepad (earlier versions being limited to files of 64KB will ask to switch to wordpad). If you have a better text editor such as one used for coding I would suggest using it (as standard notepad only supports the DOS style end-of-lines resulting in the log appearing squashed).

Text to search for;

  • "error"
  • "warning"
  • "killed himself" a suspect suicide, stuck bot? can't find path?

specific errors to search;

  • "goals" i.e, Do bots Have "No Valid Goals" In This Map
  • "invalid" i.e. Invalid Action ActivateAction
  • "aim action" i.e. Aim action screwed up, bad action_links
  • "This node has the max # of links" i.e. tried to add a 5th connection to that node

Also any special debug announcements you have added or any cvars etc. that are output from the .cfg file.

Some better editors will even count the number of times a word in the log file appears.

Balance Tests on a 9 map-cycle campaign

Balance Testing with Campaigns using cvars

Campaign Cvars;

ET does track the teams' wins so it can add the icons to the world map. The campaign cvars used are g_axiswins and g_allieswins. Now contrary to some web docs the variables are not counters but packed bitflags, where the position of the bit corresponds to the number of the map in the campaign. For instance g_axiswins = 5 and g_allieswins = 2 means axis won 2 maps and allies won 1 map. How so? well 2 = 010 binary and 5 = 101 binary. Now 101 binary in this case means map 1 and map 3 in the campaign (first and third), and 2 or 010 binary means the second map in the campaign. OK so this means that you can comeback and through the console review who won what without waiting for the game intermission (which is useless anyway on a mono-map campaign). But this method has limitations; i) It only works with campaigns. (some maps are noted as problematic for campaigns). ii) Campaigns are limited to 10 maps, after that the variables will get reset. iii) Campaigns don't support devmap/cheats including /timescale. iv) How do you know when to come back to check the progress? v) The cvars only record the number/order of wins without context (How, what , when etc.).

Now making use of the logging capability of ET can partially address two of these issues (#iv and #v). How to set ET to generate the logs is covered in the section above. But in order to get the values of g_axiswins and g_allieswins into the console log you have to include those commands in the fritzbot/configs/mapname.cfg file (replace mapname with the name of the map under test (MUTT)). That way each time the map is loaded the win values are posted to the console. Of course this means that the results for the last map played in the campaign are lost, having been reset when the campaign restarts.

//Filename: mapname.cfg
g_axiswins
g_allieswins
currentTime
//whatever else you need for this map when it loads...


Part (1): Running the MUTT in a test cycle

Making a Campaign File to test with; Campaign files are easy to make and there is a lot of help on the web like; Map rotations & Campaigns. Now to keep the test simple it should be a mono-map campaign (only one mapname used in the campaign for all map positions).

{
	name            "mapname_10"
       shortname        "mapname_10" 
	description	"mapname only 10-cycle:  For testing team-balance on mapname waypointing"
	maps		"mapname;mapname;mapname;mapname;mapname;mapname;mapname;mapname;mapname;mapname" 
	mapTC		374 374
	type		"wolfmp"
}

Explanation:Replace the 14 occurrences of mapname in the example above with the name of the map to be tested (same as the filename of the bsp file inside the etmain pk3 file. Now the shortname field should be a unique name not used in any other campaign file for best results. Fields mapTC and type do not need to be changed. When you have edited the file save it to the \fritzbot\script directory using some unique filename like mapname_10_test.campaign (replacing the mapname as you did inside the file).

Now here is one way to run a 9 map cycle and prevent the campaign auto-restarting (and reseting the cvars)

{
	name            "mapname_9stop"
       shortname        "mapname_9stop" 
	description	"mapname only 9 cycles then stop:  For testing the team balance on mapname waypointing"
	maps		"mapname;mapname;mapname;mapname;mapname;mapname;mapname;mapname;mapname;nomap" 
	mapTC		374 374
	type		"wolfmp"
}

Explanation:In this second case after replacing the 13 instances of mapname with the MUTT name, saving it and running the campaign, ET will fail at the tenth map "nomap". Since there is no map named nomap, ET fails to load it and remains in the stats display for the 9th map. Opening the console you can then see that the console repeatedly complains about not finding "nomap.bsp". Now Type \disconnect to return to the ET start page. There from the console you can then query for the g_axiswins or g_allieswins without all the spam of not finding the nomap.bsp. Write down the values and convert them manually to binary to count the wins.

To start the campaign cycle you need to join the game at the warmup of the first map. That will let the bots join. You can then either zombie or spectate, allowing the bots to play un-molested. If the team you joined has a spawn that comes under enemy fire (e.g. Battery), or has a restricted opening or path then you should not zombie, to avoid possible biasing of mobility or XP. (Don't forget to set the bot file to even teams (link at bottom)).

Part (2): Interpreting the campaign cvars

Table for interpreting g_axiswins or g_allieswins Values
Value Binary # of

Wins

What did we win? and other comments...
1 000000001 1 Won Map 1 only
2 000000010 1 Won Map 2 only
3 000000011 2 Won Maps 1 and 2
4 000000100 1 Won Map 3 only
5 000000101 2 Won Maps 1 and 3
6 000000110 2 Won Map 2 and 3
7 000000111 3 Won Maps 1 to 3
8 000001000 1 Won Map 4 only
9 000001001 2 Won Maps 1 and 4
10 000001010 2 Won Maps 2 and 4
... ... ... ...etc...
511 111111111 9 Won Maps 1 to 9

How to convert to binary using the Windows calculator app. Chose Scientific Mode, then enter the g_axiswins/allieswins value and select the radio button "bin". The binary number that appears is ordered from RIGHT to LEFT for win flags map(0) to .. map(n) where n<=15.


Permanent link to this page

Retrieved from "Unattended_Testing_Methods"