icrash(1M)
icrash - IRIX system crash analysis utility
As shipped in IRIX 6.5.7. Last changed in IRIX 6.5.7.
NAME icrash - IRIX system crash analysis utility SYNOPSIS icrash [-e cmd] [-f cmdfile] [-n bounds] [-r] [-v] [-w outfile] [-S] namelist corefile DESCRIPTION icrash is a hands-on utility that generates detailed kernel information in an easy-to-read format. icrash also provides the ability to generate reports about system crash dumps created by savecore(1M). Depending on the type of system crash dump, icrash can create a unique report that contains information about what happened when the system crashed. icrash can be run on live systems or with any namelist and corefile specified on the command line. namelist contains symbol table information needed for symbolic access to the system memory image being examined. The default namelist is /unix, which is used when analyzing a live system. If the memory image being analyzed is from a system core dump (vmcore.N.comp), then namelist must be a copy of the unix file that was executing at the time of the crash (unix.N). corefile is a file containing the system memory image. The default corefile is /dev/mem, which provides access to system memory when analyzing a live system. corefile can also be a pathname to a file (vmcore.N.comp) produced by the savecore(1M) utility. The icrash command has the options listed below. By default, all information is sent to the standard output, unless the -w option is used: -e cmd Specifies on the command line a set of commands that icrash runs automatically. If more than one command is specified, or if there are any command line options, then cmd must be enclosed within quotation marks. -f cmdfile Specifies a cmdfile that contains a set of commands that icrash runs automatically. -n bounds bounds specifies the numeric value associated with a particular namelist/corefile pair. -r Generates a standard report for the namelist and corefile specified. The reporting style differs slightly depending on the type of system dump being analyzed (panic or NMI). -v Prints the current version number. -w outfile Writes any generated output to outfile. -S Suppresses all output except that which comes from icrash commands. Normally used in conjunction with the -e flag. REPORT USAGE In order to generate a crash report, you must invoke the icrash command using the -r flag. For example: icrash -r namelist corefile This creates an ICRASH CORE FILE REPORT that prints to standard output. The resulting report can then be analyzed to determine what type of failure caused the system to crash and dump core. Note that you cannot use the -r flag to generate a crash report on a live system. INTERACTIVE USAGE Input during an icrash session is of the form: command [argument ... ] where command is one of the icrash commands described in the COMMANDS section of this reference page and argument is qualifying data that determines the specific behavior of the icrash command, which kernel structure to print, the number of items to display, etc. The following options are available to all icrash commands wherever they are semantically valid (see the COMMANDS section below). -a Display all entries -- even those that are currently unallocated, on a free list, or have a reference count of zero. -f Display additional (full) information for a structure. Note that information from all of the fields will not necessarily be displayed. Only those fields that are deemed to be most useful to the debugging process. The icrash print command can be used to display the contents of entire kernel structures. -n Follow the links of related kernel structures to their logical conclusion. For example, when used with the stream command, information for the stdata structure for the stream is displayed, followed by information on each of the queue pairs on the stream. -w outfile Redirect the output of a commands to the named outfile. The output from all icrash commands can be piped to various IRIX commands in the following manner: command [ argument ... ] | shell_command Depending on the context of the command, numeric arguments are assumed to be in a specific radix. Counts are assumed to be decimal. Addresses are always hexadecimal (with or without a leading 0x). Table address arguments larger than the size of a table are interpreted as hexadecimal addresses; those smaller are assumed to be decimal slots in the table. COMMANDS Below is a list of the interactive commands that are included with icrash. Please note that this list can change from one release to the next. To get an on-line list of all the commands available within icrash, use the help command. addtype namelist [symbol_list] Read in all the type definitions from namelist. The object file with the desired type definitions in the namelist must be compiled with debugging flags turned on (-g). If a data type of a particular name is already know by icrash, then icrash will not load that type. Note that caution should be used when intermixing 32-bit object namelists with this command when debugging 64-bit vmcores and viceversa. anon [-a] [-f] [-n] [-w outfile] anon_list Display the anon structure located at each virtual address included in anon_list. If the -a option is specified, the pfdats in the pcache is displayed in ascending order of their page numbers. Use the -f option if the command hangs when used with the -a option. The -f option displays all the pfdat's in the pcache on a "first-find" basis. anontree [-a] [-f] [-n] [-w outfile] anon_list Display the anon tree structure for each virtual address included in anon_list. avlnode [-f] [-n] [-w outfile] avlnode_list Display the avlnode structure located at each virtual address included in avlnode_list. If the -n option is specified, the root of the avltree is identified (using the parent pointer contained in the avlnode structure) and all the avlnodes in the tree are displayed. base numeric_value ... Display a number in binary, octal, decimal, and hexadecimal. A number in a radix other then decimal should be preceded by a prefix that indicates its radix as follows: 0x, hexadecimal; 0, octal; and 0b, binary. config [-f] [-w outfile] [cpu_list] Display the board configuration for a system using the information contained in the hwgraph. ctrace [-a] [-f] [-w outfile] [cpu_list] Displays a stack trace for each CPU included in cpu_list. If cpu_list is empty and defkthread is set, ctrace displays a stack trace for the cpu that defkthread was running on. If defkthread is not set, ctrace displays a trace for the cpu dumpproc was running on. If there isn't any dumpproc, ctrace displays a stack trace for the cpu that initiated the dump (based on the SP saved in dumpregs). Or, if the -a option is included in the command line, ctrace displays a stack trace for each cpu in the system. curkthread [-f] [-w outfile] Display the process, sthread, or xthread entry for the kthread currently running on all CPUs. defkthread [-w outfile] [kthread] Set the default kthread if one is indicated on the command line. Otherwise, display the current value of defkthread. When icrash is run against a system core dump, defkthread gets set automatically to the kthread that was active when the system panic occurred. When icrash is run on a live system, defkthread does not get set. The defkthread value is used by icrash in a number of ways. The trace command will display a stack trace for the default kthread if one is set. Also, the translation of certain kernel virtual addresses (primarily those contained in the kernelstack) depend upon defkthread being set. die [-w outfile] [die_addr] Print out a DWARF information entry at a given die_addr. dirmem [-b|-o|-d|-h] [-s] [-m] [-p] [-e] [-r low[-hi]] addr Display contents of the directory memory. The memory address addr can be given as a kernel symbolic address (example could be 'dirmem fork', or a hexadecimal address such as 'dirmem 0xa800000f47160000'. The leading '0x' can be omitted, the address will be understood to be hexadecimal. dirmem -h|-help Display more expanded help info. dis [-f] [-w outfile] dis_addr [count] Display the disassembled code located at dis_addr for count instructions. The default count is 1. dump [-d] [-o] [-x] [-B] [-D] [-H] [-W] [-w outfile] addr [count] Display count values starting at virtual address addr in one of the following formats: decimal (-d), octal (-o), or hexadecimal (-x). The default format is hexadecimal, and the default count is 1. eframe [-w outfile] eframe_addr Display the exception frame (containing a register dump, EPC, cause register, and status register) located at eframe_addr. etrace [-f] [-w outfile] eframe_addr Display a stack trace using the PC and SP found in the exception frame pointed to by eframe_addr. The stack address is determined using the stack pointer from the exception frame. Note that defkthread must be set equal to the appropriate kthread when stack_addr is from the kernelstack or when stack_addr is the CPU interrupt stack and there is a kthread running on that CPU. file [-a] [-f] [-n] [-p proc] [-w outfile] [file_list] Display the file structure located at each virtual address included in file_list. If no addresses are specified, display all currently open files. If the -p option is used, display all files opened by proc. proc can be specific as a virtual address or as a process PID (preceded by a #). findsym [-f] [-w outfile] address_list Locate the kernel symbol closest to each virtual address contained in address_list. from cmdfile Read in commands from cmdfile and run them as if typed from the icrash command line. Note that the -w option and pipe (|) option are not valid options for the from command. These options can, however, can be included in the commands contained in cmdfile. fstype [-w outfile] [vfssw_list] Display the vfssw structure for each entry in vfssw_list. Entries in vfssw_list can take the form of a virtual address or vfssw table slot number. If no vfssw entries are specified, display the entire vfssw table. func [-w outfile] [func_addr] Print out information about the function located at func_addr. help [all | command_list] Display a description of the named commands, including command syntax and an overview of how to use the command. The all option displays help information for every command. history [-w outfile] Dump out the last 20 history commands. You can also use ! to access old commands (including !!, !-N, etc.) hubreg [-b|-o|-d|-x] [-s] [-n n] register Display contents of a hub register. The register name can be given in upper case or in lower case or even as a full address. If the address corresponds to a real hub register, its name will be found and displayed. The register given by its name (and not by its address), does not have to be its full name, only the first so many letters making its name unique are needed. If a unique match cannot be found, the command will list all the register names matching the characters given, and you can then pick up the correct one from the list. hubreg -l|-list List all defined register names. hubreg -h|-help Display more expanded help hubreg -listacc Display info on access mode definitions. hwpath [-f] [-w outfile] element_lsit Display all pathnames from the /hw file system that terminate with the names included in element_list, or that contain label names included in element_list. If no names are specified, display all unique pathnames. Unique pathnames are ones that terminate in a file or symbolic link rather than in a directory. When the hwpath command is issued with a -f command line option, the vertex handle for each element in the path is also displayed. Note that the hwpath command provides a view of the the internal hwgraph data structures which are are presented to the user via a filesystem-like interface. inode [-f] [-w outfile] inode_list Display the inode structure located at each virtual address included in inode_list. inpcb [-f] [-w outfile] inpcb_list Display the inpcb structure located located at each virtual address included in inpcb_list. kthread [-a] [-f] [-w outfile] [kthread_list] Display relevant information for each entry in kthread_list. If no entries are specified, display information for all active kthreads (processes, xthreads, and sthreads). Entries in kthread_list should be in the form of a virtual address. ktrace [-a] [-f] [-w outfile] [kthread_list] Displays a stack trace for each kthread included in kthread_list. Entries in kthread_list should be in the form of a virtual address. lsnode [-f] [-w outfile] lsnode_list Display the lsnode structure located at each virtual address included in lsnode_list. mbuf [-a] [-f] [-n] [-w outfile] mbuf_list Display the mbuf structure located at each virtual address included in mbuf_list. In irix6.5 the mbuf command when used with the -a option will display details of all the mbufs allocated on the system along with output similar to the "netstat -m" output on a live machine. memory [-f] [-w outfile] Display information about the memory installed on each node in the system. If the memory command is issued with the -f option, display information about the memory contained in each memory slot on each node. mlinfo [-a] [-f] [-n] [-w outfile] mlinfo_list. Display information from the ml_info struct for each virtual address included in mlinfo_list. mntinfo [-f] [-n] [-w outfile] mntinfo_list Display the mntinfo structure for each virtual address included in mntinfo_list. The mntinfo contains information about NFS mounted file systems. mrlock [-f] [-n] [-w outfile] mrlock_list Display the mrlock_s structure located at each virtual address included in mrlock_list. If the -f command line option is specified, display all queues of WAITERS and HOLDERS. If the -n command line option is specified (in conjunction with the -f option), then walk each queue displaying information on associated kthreads. nodepda [-f] [-w outfile] [nodepda_list] Display the nodepda_s structure associated with each item in nodepda_list. Entries in nodepda_list can take the form of a node ID or a virtual address pointing to a nodepda_s struct in memory. If nodepda_list is empty, information for all nodes will be displayed. outfile [outfile] Set outfile (the file where all command output is sent) if outfile is indicated. Otherwise, display the current value of outfile. pager [-w outfile] [on | off] When pager is on, all command output is piped through pg. When the pager is off, all output is printed continuously without page breaks. When the pager command is issued without a command line option, the current state of the pager is displayed (on or off). Note that when the pager is on, command output cannot be redirected to a pipe. pda [-f] [-w outfile] cpu_list Display the pda_s structure associated with each item in cpu_list. Entries in cpu_list can take the form of a CPU ID or a virtual address pointing to a pda_s struct in memory. If cpu_list is empty, the pda_s structure for all CPUs will be displayed. pde [-w outfile] pde_list Display the pde (page descriptor entry) structure located at each virtual address included in pde_list. pfdat [-a] [-f] [-w outfile] pfdat_list Display the pfdat structure located at each virtual address included in pfdat_list. If the -a flag is issued, display all of the pfdat structures on the system (one per physical page). pid [-a] [-f] [-w outfile] pid_list Display the pid_entry structure for each entry in pid_list. Entries in pid_list can take the form of a pid table slot number, process PID (following a #), or a virtual address. If no entries are specified, display all active (in use) pid entries. pregion [-a] [-f] [-n] [-p proc] [-w outfile] [-u uthread] [pregion_list] Display the pregion structure located at each virtual address included in pregion_list. If the -p option is used, display all pregions allocated to proc. When used with the -n option display all the pde information for the proc or uthread specified with the -p or -u options respectively. proc can be specific as a process PID (following a #) or virtual address. print [-d] [-o] [-x] [-w outfile] expression Evaluate expression and print the result. An expression can consist of numeric values, operators, kernel variables, typedefs, struct/union members, or a combination of above. The following are examples of valid expressions: (((2*3+4/2)*2+(2/6))/2) *((struct socket*)0xa80000001019f2c8)->so_rcv->sb_mb (((pte_t*)0xa8000000005ef088)->pte_pfn)<<14|0xa800000000000000 printd [-w outfile] expression The printd command is the same as the print command except that all integers are printed as decimal values. printo [-w outfile] expression The printo command is the same as the print command except that all integers are printed as octol values. printx [-w outfile] expression The printx command is the same as the print command except that all integers are printed as hexadecimal values. proc [-a] [-f] [-w outfile] [proc_list] Display the proc structure for each entry in proc_list. If no entries are specified, display all active processes. Entries in proc_list can take the form of a process PID (following a #) or virtual address. ptov [-w outfile] address_list Display all possible virtual address mappings (K0, K1, and K2) for each entry in address_list. Entries in address_list can be a hexadecimal physical address or a PFN (following a #). queue [-f] [-n] [-w outfile] queue_list Display the queue structure located at each virtual address included in queue_list. If the next option (-n) is specified, a linked list of queues, starting with each specified queue then following the q_next field, is displayed. quit Exit icrash. Note that q prompts for confirmation unless a ! is appended to the command line. region [-f] [-n] [-p proc] [-w outfile] region_list Display the region structure located at each virtual address included in region_list. If the -p option is used, display all regions allocated to proc. proc can be specific as a process PID (following a #) or virtual address. report [-w outfile] Print out the base report automatically generated by the savecore script when the system comes back up after a system crash. rnode [-f] [-n] [-w outfile] rnode_list Display the rnode structure for each virtual address included in rnode_list. The rnode contains information about a virtual node that acts as an interface between two NFS systems. runq [-f] [-w outfile] Display a list of processes currently on the run queue. Using the -f option will print out more specific information about the processes. sbe [-w outfile] Print out the single bit error information for a Challenge or Onyx system (including R8K and R10K systems), assuming the right single bit flags in the kernel are turned on. search [-B] [-D] [-H] [-W] [-w outfile] [-m mask] pattern [address] [length] Locate contiguous bytes of memory that match the values contained in pattern, beginning at address for length bytes. pattern consists of a string of, from one to 256 hexadecimal digits (with no embedded spaces). For full word searches (the default), the first word of mask is anded (&) with each word of memory and the result compared against the first word in pattern (which is also anded with the first word of mask). If there is a match, subsequent words in memory are compared with their respective words in pattern (if there are any) to see if a match exists for the entire pattern. If the -D option is issued, the search is conducted on double word boundaries. If the -H option is issued, the search is conducted on halfword boundaries. If the -B option is issued, the search will be performed without regard to double word, word or halfword boundaries. If a mask is not specified, mask defaults to all ones for the size of pattern. For all but string searches, the number of hex digits specified for pattern cannot be less than will fit into the memory boundary size specified. For example, patterns for halfword boundary searches, must contain (at least) eight hex digits (two per byte). Also, any extra digits beyond the specified boundary will be trimmed off from the right side of the input line. In addition to finding matches for hex values in memory, it is possible to search for strings of characters as well. Just begin and end the character search pattern with double quotes ("). The ASCII value for each character in the string will form the corresponding byte values in the search pattern. address can be specified as either a virtual address (K0, K1, K2, etc.), a physical address, or as a PFN (directly following a pound # sign). If no address is specified (or if the one specified does not map to a valid physical memory address), address defaults to the K0 address mapped to the start of physical memory. An optional length parameter specifies the number of bytes to search. If length is not specified, it will be set equal to the size of physical memory minus the starting physical address. Note that length can be specified ONLY when an address has been specified. sema [-f] [-w outfile] [sema_list] Display the sema_s structure located at each virtual address included in sema_list. With the -f option a list of waiters, waiting on the sema, are displayed. sh [command] Escape to the shell with no arguments or execute command. Note that it uses your SHELL environment variable to determine which shell to use. sizeof structure[.field] Dump out the size of structure entered on the command line. The value returned will be in bytes. The user can also use a structure.field notation in order to find out what the size of a field in structure. If the sizeof command is unable to find structure or field, then the structure information is not located in the symbol table. Also note that the sizeof command only returns the size of struct and union types. slpproc [-w outfile] Print out the set of sleeping processes on the system. socket [-f] [-n] [-w outfile] [socket_list] Display the socket structure for each virtual address included in socket_list. If no entries are specified, display all sockets that are currently allocated. If the next option (-n) is specified, a linked list of protocol control block structures associated with each socket will also be displayed. stack [-w outfile] stack_pointer Print out the stack type, stack address, and stack size for a the stack stack_pointer is from. If stack_pointer is from a kthread stack, then defkthread must be set equal to the address of the appropriate kthread. stat [-w outfile] Display system statistics and the putbuf array, which contains the latest messages printed via the kernel printf/cmn_err routines. sthread [-a] [-f] [-w outfile] [sthread_list] Display relevant information for each entry in sthread_list. If no entries are specified, display information for all active sthreads. Entries in sthread_list can take the form of a virtual address. strace [-a] [-f] [-l] [-w outfile] [pc] [sp] [stack_addr] [level] Displays all complete and unique stack traces (containing level or more stack frames) from the stack starting at stack_addr. If level isn't specified, then each stack trace must have at least five frames to be considered valid. Alternately, use a specific pc and sp to generate a stack trace from the stack starting at stack_addr. Or, when the -l command line option is specified, displays a list of all valid kernel code addresses contained in the stack starting at stack_addr, along with their location in the stack, source file and line number. Or, if the -a option is specified, display ALL traces of level or more frames, including invalid traces and duplicate (sub) traces. Note that defkthread must be set equal to the kthread being analyzed when stack_addr is the kernel stack address or the CPU interrupt stack and there is a process running on that CPU. This is necessary so that the trace can automatically "jump" from the first stack to the kernel stack of the process being analyzed. stream [-a] [-f] [-n] [-w outfile] [stream_list] Display the stdata structure for each virtual address included in stream_list. If no entries are specified, display all streams that are currently allocated. If the next option (-n) is specified, a linked list of queues that are associated with the stream is also displayed. strings [-w outfile] start_address | symbol [count] Display count strings of ASCII characters starting at start_address (or address for symbol). strstat [-f] [-w outfile] Display information about streams related resources (streams, queues, message blocks, data blocks, and zones). struct [-f] [-l] [-n] [-w outfile] struct addr Print structure information for struct using the block of memory pointed to by addr. swap [-f] [-w outfile] [swap_list] Dump out the list of swap devices, including the vnodes that are represented. The number of pages, number of free pages, number of max pages, priority, and device are listed. The -f flag dumps out the name of the swapinfo entry as well. symbol [-f] [-w outfile] symbol_list Displays information about each kernel symbol included in symbol_list. tcp [-f] [-w outfile] tcpcb_list Display the tcpcb structure for each virtual address included in tcpcb_list. tlbdump [-w outfile] [cpu_list] Display TLB information for each CPU indicated in cpu_list. If no CPUs are indicated, TLB information for all CPUs is displayed. trace [-a] [-f] [-w outfile] [kthread_list] Displays a stack trace for each kthread included in kthread_list. If kthread_list is empty and defkthread is set, then a stack trace for the default kthread is displayed. If defkthread is not set, then a trace will be displayed for the process stored in dumpproc. If there isn't any dumpproc, then trace displays a trace for the cpu that generated the core dump. type [-w outfile] [symbol] | type_addr Display the contents of memory starting at addr using the type definition type to format the output. Structures, unions, etc. are displayed in a manner similar to the way they were originally defined. The values contained in the various struct/union members are displayed in accordance with their type declarations. For example, kernel addresses are displayed as hexadecimal values, signed and unsigned integers are displayed as decimal values, etc. Struct/union members that are themselves structs or unions are expanded in the much the same way. Alternately, if a kernel symbol is entered, instead of type_addr, an attempt is made to determine the type of the symbol. If the type can be determined, it will be displayed as described above. unpcb [-f] [-w outfile] [unpcb_list] Display the unpcb structure for each virtual address included in unpcb_list. uthread [-l] [-f] [-n] [-k] [-S] [-w outfile] uthread_list Display relevant information for each entry in uthread_list. If no entries are specified, display information for all active uthreads. When the -S command line option is specified, displays a list of all siblings for the specified uthread. Entries in uthread_list can take the form of a process PID (following a '#'), or virtual address. Note that with the PID option, all uthreads that share a specified PID will be displayed. When the -k command line option is specified, display all uthread_s structs as kthread structs. When -l is used, only the uthreads contributing to the load average are shown. vertex [-a] [-f] [-w outfile] [vertex_list] Display the graph_vertex_s structure located at each virtual address included in vertex_list. vfs [-f] [-w outfile] vfs_list Display the vfs structure for each virtual address included in vfs_list. vnode [-a] [-f] [-n] [-w outfile] vnode_list Display the vnode structure for each virtual address included in vnode_list. vproc [-a] [-f] [-n] [-w outfile] vproc_list Display the vproc structure for each virtual address included in vproc_list. vsocket [-a] [-f] [-n] [-w outfile] vsocket_list Display the vsocket structure for each virtual address included in vsocket_list. vtop [-w outfile] address_list Display all possible virtual address mappings (K0, K1, and K2) for each virtual address in address_list. walk [-f] [-l] [-s] [-w outfile] [struct field|offset addr] [addr offset size] Walk a linked list of kernel structures or memory blocks. The walk command has three modes of operation. By default, output from the walk command consists of a linked list of formatted structure entries. Each entry contains a single line of output, similar to the output of other icrash commands. The list of structures which can be displayed in this manner is limited. To see a listing, issue the walk (or struct) command with the -l command line option. Note that when viewing the list of structures, only those structures marked YES in the LINKS column contain links that can be followed. When the walk command is issued with the -s option, each structure is displayed, in its entirity, in a C-like format. The only limitation on which structures can be walked in this manner is that structure related information must be contained in the kernel's symbol table. Even if information about a structure is not available, it is possible to do a hex memory dump of each structure in the list. With the first two options outlined above, the structure name, field name (or byte offset), and next pointer are required. With the third option, a start address, byte offset, and struct size are required. Examples: walk xfs_inode i_next a800000000345678 walk xfs_inode 8 a800000000345678 walk -s mrlock_s mr_queue a80000000345678 walk -s mrlock_s 8 a80000000345678 walk a800000000345678 8 40 The last example illustrates the use of the walk command to display debugging information about a structure even if its type information is not found. It does a memory dump of each structure in the list. whatis [-a] [-f] [-l] [-w outfile] expression Display type specific information for expression (see the print command for what constitutes a valid expression). If the -a option is specified, display a list of all structures, unions, typedefs and variables. When, in conjunction with the -a option, the -f option is specified, expand all struct and union definitions to show information for each field. xthread [-f] [-n] [-k] [-S] [-w outfile] xthread_list Display relevant information for each entry in xthread_list. If no entries are specified, display information for all active xthreads. Entries in xthread_list can take the form of a virtual address. When the -k command line option is specified, display all xthread_s structs as kthread structs. zone [-a] [-f] [-w outfile] [zone_list] Display information about zone memory allocation resources. If the -a option is specified, display a list of all zones allocated in the system by zone type. ? [-w outfile] Displays a list of available commands. NOTES Each version of icrash is specific to the OS release that it came from and does not necessarily work on any other OS release. Do not copy icrash to any other IRIX system unless the OS versions are identical (including patch levels). Running icrash on a live system can sometimes generate random results, as the information being viewed is volatile at the time it is displayed. When running against compressed system crash dumps, an index file will be created to allow icrash quick access to compressed pages inside the dump. In addition, if an index file exists, icrash will start much more quickly against a crash dump. icrash will always try to write the index file to the same directory where the system crash dumps are stored. If the directory is not writable, you will see the message: icrash: save of index "filename" failed: permission denied where filename is the name of the index file that icrash tried to create. To avoid this, users can create symbolic links to the namelist and corefile in a writable directory, and run icrash on the symbolic links. The index file will then be created in the writable directory where the symbolic links are located. The GNU readline library is used by icrash in interactive mode. Please see the GNU General Public License available from the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. BUGS & ISSUES Some icrash commands will not work completely on O2's with the R10000 cpu in them. This is because kernel space in the address range 0x800000- 0x9fffff are not readable to user processes for this hardware configuration. The commands that are most affected are the "pfdat" and "zone" commands.