Chapter 10 Converting from C-Index/PC

C-Index/II is a superset of C-Index/PC and C-Index/Plus. Programs and files created using those libraries may be converted for use with C-Index/II. Comments made in this chapter about C-Index/PC also apply to the C-Index/Plus product, unless specifically noted otherwise.

C-Index/II includes many new features which are not in C-Index/PC. The following list details the features which have changed.

The use of 16 bit pointers in C-Index/PC to locate nodes has been replaced with 32 bit pointers. Nodes now are defined internally by the typedef of NDEPTR. Using the default 1K node size allows the maximum file size to be 8.7 Terabytes (trillion bytes). To achieve a file size in excess of 4 Gigabytes (the limit for Unix style file addressing) will require changing the fileSeek function as required by the operating system. Although large files of this type are possible, it is unlikely that the maximum will be reached because of the length of time that it takes to build an index of this size.

The larger maximum file size has required changing all the file header and node header information to 32 bits.

The file header has been changed to eliminate the word boundary alignment problem when using different compilers and compiler switches. The header includes space for "custom" binary information, which is essentially a single 100 byte record that can be written and read directly from the header. The header now has an array of flags indicating the type of key stored in each index. Also, the header has a field for the number of records stored in the file and the number of deleted record numbers stored in DELINDX (used internally by the multi-key routines).

Word order alignment on the record number is taken care of automatically by the implementation of the macro trnsrec. Entries in the index nodes store the node number in the same location as data in the leaf node, that is, after the key value.

The length of the key is determined according to the type of key stored in that index (which is indicated in the file header). For string keys (the default type), the length is the length of the string including the zero terminator. For numeric keys, the length is determined by the compiler sizeof() function. The int data type is either 2 or 4 bytes (depending on the compiler), shorts are 2 bytes, longs are 4 bytes, floats and doubles are dependent on the compiler. Binary, segmented and custom keys have a one byte length value as the first byte of the key.

The length of the data part of the record is calculated internally by taking the length of the entire entry, subtracting the length of the key and the length of the overhead (7 bytes).

A general purpose utility program, convert, is provided to read the file in C-Index/PC format and write it out in C-Index/II format. The command line invocation is as follows:

The error file will contain any records that do not pass the ccheck step of the conversion (the same usage as in the rebuild utility).

A new function, cconvert has been added for converting files from within an application:

This function works in a similar manner to the cbuild function. The old format file is opened, the new and error files are created, then each entry in the old format file is added to the new file by reading each node sequentially and adding its contents to the new format file. After all nodes in the source file have been converted, the ccheck function is used to check the integrity of the file.

C-Index/II has the ability to re-use nodes that have been made empty through deletion of keys. For example, this allows the program to add any combination of keys, delete all of the keys, and then re-add any other combination of keys without any wasted space being left in the file.

When files are created or opened, the default mode of the file is to not perform full deletion. To force the system to perform full tree deletion, change the psp variable fulldel to TRUE (non-zero).

C-Index/II source code uses a more conventional method for building and maintaining the libraries. The three large files have been split into smaller files, and a set of make files manages regenerating the system when changes are made. Also, the compiler and operating system specific code exist in a single code file and header file, instead of the unusual two files that serve as both code and header in C-Index/PC. This will simplify porting C-Index/II to new compilers and operating systems (see notes below about porting).

Each compiler specific directory contains a set of batch or command files for that specific compiler. These files invoke the necessary make files to regenerate the source code for that compiler. For example, the batch file for recompiling libraries is makelibs.bat.

Additional compilers and operating systems may be added to the C-Index/II package. Be sure to check the readme.doc file on the source code disk for the latest information.

Provision has been made to read and write files on a machine with an architecture different than 8086 type machines. This involves changing the byte order of shorts, ints, and longs as required by the specific machine reading the file. For example, 68000 computers and IBM 370 mainframes use the high to low value byte ordering, whereas Intel family processors and VAX computers use low to high value byte ordering. With this change to the code, data files can be created using any computer and read on any computer, provided that the application uses only data fields that are compatible between different machines.

C-Index/II single-key routines support index key values other than string keys. The default value for indexes is the string type (STRINGIND). No special steps need be taken to specify string type keys. To use any other type of key, the application program must indicate to C-Index the index key type, either when creating the file or by using the cchgndx() function. Key types are indicated to C-Index/II functions using the NDXLIST data structure. For more information, refer to the notes on bcreate and dbcreate.

The bcreate and dbcreate functions have been added to provide control over file mode and index type. The existing file create functions have been modified to default to creating files which are in the Intel file format (low to high byte order) with all indexes set to type string (STRINGIND). The new functions specifying the format of files to be created are:

The header has a new variable, wordorder. This header value also is stored as the variable "wordorder" in the Parameter Structure. The binary value of 1 is stored in it as a 32 bit word. If INTELMODE has been selected, the numeric value of 1 will be stored in low to high byte order (Intel format). If NONINTELMODE has been selected, the numeric value of 1 will be stored in high to low byte order. NATIVEMODE will cause the numeric value to be stored in the native mode of the computer creating the file. C-Index/II automatically uses this value to determine which format is contained in the file.

The index list is used to specify the types of key values the system will be processing in each index. The list is a structured array defined as follows:

The list only needs to indicate indexes that are not of type STRINGIND (ASCIIZ strings). If all indexes are strings, the indexlist parameter of bcreate and dbcreate can be specified as NULL. The keyn value indicates which index will have a modified index type. The keytype value indicates the type of key:

STRINGIND ASCIIZ string keys.

NOCASEIND ASCIIZ string key, ignore case

BINARYIND Binary keys.

SHORTIND Short integer keys (16 bit).

INTIND Integer keys (compiler specific).

LONGIND Long integer keys (32 bit).

FLOATIND Float keys (compiler specific).

DOUBLEIND Double keys (compiler specific).

SEGIND Segmented keys.

CUST1IND Custom key #1 (customer supplied).

CUST2IND Custom key #2 (customer supplied).

CUST3IND Custom key #3 (customer supplied).

CUST4IND Custom key #4 (customer supplied).

CUST5IND Custom key #5 (customer supplied).

The segptr value is a pointer to the segment description list for segmented keys (SEGIND). For all other key types, this value must be NULL. See notes in the section covering segmented keys.

The indexlist controls the definition of index types at the time of file creation (using bcreate). When the file is subsequently opened, an indexlist must be supplied to bopen which matches the current definition of indexes (or NULL for all-string indexes).

The new read mode parameter allows the specification of read-only access for the opened file. This solves problems for some LAN systems where users need to be restricted to read only access for specific files. The bopen and dbopen functions replace the mopen and dmopen functions of C-Index/PC. There are macros in CNDX.H that translate the old names to the new names with a default of CRDWRITE.

When opening a file for read-only, it is possible to receive a BADWRITE (-33) error message. This indicates a previous process locked records in a file (while using the file in read/write mode) and then terminated without closing the file. As a result, C-Index is attempting to automatically remove invalid lock semaphores. The solution is to open the file in read/write mode to clear the errant semaphores.

The indexlist parameter must be specified for all files that have been created with non-string keys (using a create function or cchgndx function). For files where all indexes are string type (the default), the indexlist parameter should be NULL. Refer to the bcreate function description for details on the indexlist definition.

When a file is opened in shared mode in C-Index/PC, the cnext, cprev, and dseq functions must always simulate sequential access by repositioning the current pointer with the cfind function. C-Index/II improves this by requiring repositioning only in the event that the file has been changed (as evidenced by the update counter in the header).

With changes to the source code, the node size may be changed to any arbitrary size that is at least 1K. The typical application of this feature is in CDROM access where having larger nodes improves access time from the extremely slow optical disk media. To change node size, change the define statement in the library header file which reads as follows:

When C-Index/II records are updated using dupdate, the record number does not change. This is the current behavior for C-Index/PC. In C-Index/Plus the record number is changed with each update. This will eliminate problems found in C-Index/Plus with multi-user access to records that are being updated and will allow the application program to depend on the record number as a consistent way to retrieve a record.

Some modifications of memory allocation have been made to reduce overhead for current key information. C-Index/PC uses variables declared in the parameter structure for all the possible indexes, which takes about 2K for 20 indexes. In C-Index/II the space for storing current key information is allocated only when an index is used by the program, and released when the file is closed. The new allocation code is designed to work correctly with Windows and Mac memory allocation requirements via operating system specific interface code that can easily be modified to meet custom methods of memory allocation. If an error occurs in allocating memory (out of memory condition of any type), the error code OUTOFMEM will be returned.

In addition, the virtual disk buffers are allocated from heap space. Compatibility is provided for mixed model memory where the heap is in far space. The code will also work where the buffers have been allocated from movable heap space, such as with Windows and Macintosh systems. The default number of buffers is 10. To modify the number of buffers, call the cibufnum function or set the global variable "bcbnum" to the number of buffers desired before opening any C-Index files. For example, to have 30 buffers:

The C-Index/PC method of adding keys does not provide for optimal speed when adding keys known to be in sorted order. C-Index/II uses fewer key comparisons for insertion by employing a modified binary search of the index nodes instead of a linear search. This new technique results in approximately 3 times speed improvement for sequential key insertion and will also improve performance for non-sorted key insertion and key retrieval.

C-Index/II detects when adding a key to the end of the file (after last entry of last index, usually DATAINDX or SEMAINDEX) and splits the node to improve storage usage. C-Index/II also can create separate root nodes for each index so that this split mechanism will work for every index in the file, not just the last index in the file.

The Dump utility has been improved with the addition of several new features. Refer to the C-Index/II Utilities chapter in this manual for details.

The Compact utility no longer performs the Goodbad Utility types of checks on the input file. It has also been modified to compact files with custom header information and binary keys.

Two new single-key routines have been added to assist in keeping track of the number of records that are stored in the file. These primitives are used by the Multi-Key routines to maintain the number of active records in a file.

The csetcnt function sets a value in the header which can be read using the cgetcnt function. The cgetcnt function returns the record count value in the psp->rec variable.

The Multi-Key routines now automatically keep track of the number of multi-key records stored in the file. The new multi-key function, dgetcnt returns the active record count in the psp->rec variable.

Two new functions allow the application to read and write a binary block of information into the header. The maximum number of bytes that may be stored in the header is defined by MAXCUSTINFO (default value of 100).

The length of the data specified when setting custom information is stored in the header along with the data itself. When using cgetcust, the datalen parameter is the size of the buffer to use for returning the data. If the buffer is too small for the data being read, the BUFOVER error will be returned. Whatever the size of the return data buffer, the number of bytes actually returned will be in psp->retlen and the size of the information in the header will be returned in psp->dlen.

The default number of indexes in an index file (MAXINDXS) is 20. The last 5 indexes are reserved for internal use by C-Index/II and should not be used by single-key or multi-key routines from within the application. This allows for 15 indexes to be used by the application. This maximum in defined by the macro MAXDINDX. The number of indexes may be increased or decreased by changing the MAXINDXS define in the CNDX???.H compiler specific header file, and recreating the libraries. Because most of the space for keeping track of the current index position is allocated only when an index is used, this large number of available indexes will not significantly impact memory usage unless all of the indexes are actually used. Each default index requires 5 pre-allocated bytes of space (a 4 byte pointer and a 1 byte flag).

Index key types may only be set at file creation or by using the cchgndx function:

An index must be empty when its key type is changed. If any indexes are changing type but already have entries, an error INDEXIST will be returned by cchgndx.

Files with non-string index keys must be opened with the bopen or dbopen functions and the indexlist specified must match the type saved in the file header. If a mismatch is detected, the BADINDTYP error will be returned when attempting to open the file.

The parameter structure value key and key pointer function parameters have been redefined as KEYPTR (typedef for void *) to allow the key to conveniently point to any type of data. C-Index/II handles the key information based on the definition of the key type in the file header. If the pointed to key value is of the wrong type (string instead of binary, for example,) unpredictable results will occur since there is no way for the system to ensure the pointer points to a key value of any particular type.

This allows the user to implement any type of custom key value and collating sequence. The function of the custom key values must be added to the compiler specific source file (for example, cndxmsc.c for Microsoft C/C++ under MSDOS). The functions that must be modified are cikeycmp and custincrkeyval. If no special key comparison is supplied, the standard unsigned byte by byte comparison will be used by default. The format of the custom key entry must begin with a one byte length count. Otherwise, the action of the custom key comparison is entirely under the control of the code defined by the user.

The semaphore key values are now defined as binary keys where the first byte is the length of the semaphore value, the second byte is the index number where the entry to be locked appeared, and the rest of the key is the key value to be set. Because semaphores only match on exact key values, the binary key comparison is compatible with locks on any type of key entry. If your application uses the semaphore routines (msetsema, mtstsema, mclrsema), it will require modification to create the correct binary semaphore value.

When attempting to rebuild a corrupted file, if the index types in the header are invalid, the rebuild program will not work correctly. Also, rebuild will not work with segmented keys, since the segment information is not available. The cbuild function has an additional parameter for the pointer to an indexlist. If a NULL value is specified, the header from the source file will be used. Otherwise the indexlist supplied will be assumed to be correct for the input and output files. By revising the rebuild utility to include your specific datalist, you will be able to more reliably rebuild C-Index/II files and you will be able to rebuild segmented key files.

Many new error codes have been added. A list of error codes is supplied in the C-Index/II Reference Guide. The most up-to-date error code information is in the C-Index/II include file cndx.h. The following is a list of some new error codes used in C-Index/II:

The Multi-Key functions now allow you to specify any field type as being a key. The keytype field of the datalist supports all the key types allowed in the indexlist specification (see notes on dbcreate):

For compatibility with C-Index/PC you may continue to use the STRINGFLD type.

Notes: The segmented key value must be specified as a field type of SEGFLD. The custom key values must be a field type of BINARYFLD.

A new function, "dsetkey", has been added to simplify creating a key value for use with dfind. The format of the binary and segmented keys passed to dfind is the same as for the single-key cfind function: a one byte unsigned length followed by the specified number of unsigned bytes. The dsetkey function returns the correct key value based on the index number and a datalist. Parameters are as follows:

The function returns the standard C-Index/II condition code. The parameter structure of the file to be used for the dfind operation is specified. The keyn parameter is the index number to be specified for the dfind function. The datalist would typically be the same as used for the dread operation. The keybuf is a buffer where the key will be built and must be large enough for the key. Specifying a character array of MAXKLEN will always be large enough. Before calling dsetkey the field values required for the key must be stored in the field locations pointed to by the datalist. The dsetkey function then reads these values and builds a key. For most types of keys, it is not useful to call this function, however, for binary and segmented keys, it simplifies the process so that you do not need to be concerned with internal representations of the search key.

When accessing files opened in SHARE mode, the dupdate, ddelete, cchgcur and cdelcur functions now require that the record be locked before performing the update or delete operation.

A new flag, "nolock", has been added to the psp that turns on and off the requirement of locking records before delete and update. The value of the flag is:

The default action of C-Index/II (and C-Index/PC) is to reassign record numbers of deleted records to newly added records. This was particularly helpful in C-Index/PC to reclaim unused space caused by deleted records. In C-Index/II, this is not needed when full tree deletion is turned on.

A new flag, "reuse", has been added to the psp that turns on and off the reassignment of record numbers in the multi-key routines. The advantages of turning off record number reassignment are:

C-Index/II provides for specifying segmented keys in single-key and multi-key routines. The segmented key feature allows an application to create a key that is made up of several fields (segmented keys are also called concatenated keys in some data managers). C-Index/II allows the keys to be of differing lengths and types. Segmented keys are variable in length. There are no padding bytes added to the key to make the individual segments position to fixed locations. Instead, C-Index/II keeps track of each segment separately and stores only the number of bytes needed for that value. For more information about segmented keys, refer to the Advanced Key Usage chapter in this manual.

A new function, endbcb, releases buffer space that has been automatically allocated. Some systems require that you release all allocated space before exiting the program. The endbcb function is the inverse of the initbcb function that is automatically called when the first file is opened. The compiler specific source file contains the function cideallocbuf to perform the actually freeing of space. The function prototype is:

All files must be closed before calling this function. If the files are not closed, the function will return FAIL. The parameter structure pointer passed to the function must have been initialized with compiler specific information, if any. For example, MSDOS and OS/2 environments do not require any initialization, so this is a pointer to any uninitialized parameter structure space. Under other operating systems, the structure may require additional information (via the portinfo pointer). With the supported compilers in the release, no initialization is required.

The compact, goodbad and rebuild utilities will not work with files that have been created with non-string or segmented keys. This is because these utilities need to know something about the keys being processed. They work correctly when the index uses only string keys. If your application needs any of these utilities for files which include non-string and/or segmented keys, you will need to modify the code to work with the types of keys being used.

C-Index/II is almost completely program compatible with C-Index/PC and C-Index/Plus. A new file structure is used that will require converting existing files to the new format using the supplied utility programs and function calls (see notes above). In order to convert existing C-Index programs for work with C-Index/II the following changes will need to be made.

In place of CINDEX.H and CINDEXSB.H, the application must include CNDX.H. This in turn includes a compiler specific header file, such as CNDXMSC.H in the case of Microsoft C or CNDXTC.H in the case of Borland C. Both CNDX.H and the compiler specific header file must be accessible to the compiler by moving them to the application directory or adding an include file directory command to the application module.

The compile command line for the application must have a define for the type of compiler and operating system being used so that the correct compiler specific header file will be included. For example, the compile line for the example program (compiled using Microsoft C to run under MSDOS) contains the defines:

to specify the Microsoft C compiler and the MSDOS operating system. Consult the readme.doc file supplied for the C-Index/II version you are using for more information specific to supported compilers.

All previous Microsoft C versions of C-Index require the use of the -Zp option. The make file and pre-compiled libraries supplied with C-Index/II do not use the -Zp flag. You can recompile the library with or without the -Zp flag and the structure of the C-Index/II header file will not be affected. If you have previously modified C-Index/PC libraries to not use -Zp, you will have files that are incompatible with the conversion utility and conversion function call. This will require that you make changes to the cconvert.c source code before converting the files or run the header conversion utility before converting the rest of the files.

The number of buffers is not changeable in the C-Index/II header file. Instead, if you need to modify the number of buffers, the application must call the cibufnum function or set the global variable bcbnum to the number of desired buffers before any C-Index files are opened.

Unlike C-Index/PC, C-Index/II uses memory allocation to store information about the current index pointer location. This saves considerable space in the parameter structure but increases the space required in the heap. For applications that are short on memory, this may require changes to ensure that the heap will be large enough. Running out of heap space during any C-Index operation will result in the return of the OUTOFMEM error.

If the parameter structure has been allocated off of the stack as an automatic variable, the stack may not need to be as large as before. However, it is advisable to be sure the application stack is large enough, since stack bugs are difficult to detect and fix.

Applications using C-Index/Plus that depend of the fact that record numbers change when performing dupdate will need to be modified. C-Index/II is like C-Index/PC in that the record number assigned to a multi-key record is not changed when performing dupdate.

If your application uses the msetsema, mtstsema, or mclrsema functions, you will need to change your functions to create binary semaphore keys, instead of string semaphore keys. For more details consult the chapter in this manual "Advanced Key Types".

Some previous versions of C-Index have not required that a multi-key record be locked before preforming dupdate or ddelete. C-Index/II does require such a lock when the file is open in SHARED mode. If your application does not lock records in SHARED mode, either add record locking calls or turn off this locking requirement by setting the nolock parameter structure variable:

When using C-Index with other add-on libraries, conflicts between definitions can occur between the packages. For that reason, C-Index/II has redefined names which have been found to be in conflict with past versions of C-Index. By commenting out the define NEW_FLDTYPES define in the compiler specific header the following old names are re-enabled:

C-Index/II does not allow parameter structures to be moved in memory. Some applications running under Windows using C-Index/PC save space in the static segment by having a single parameter structure in the static segment, and then copying each open parameter structure from far memory. This technique is not compatible with C-Index/II. You must have all open parameter structures in the static or stack segment at all times. Otherwise corruption of the index files may result from C-Index becoming confused about which buffer in memory belongs to which file on disk.

C-Index/II Home Page

www.triosystems.com © Copyright 1996 - 1999 Trio Systems LLC

C-Index/II User Guide © Copyright 1983-1997 Trio Systems LLC

User Guide Revision Date: 5/2/96

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of Trio Systems LLC.

Trio Systems is a registered trademark of Trio Systems LLC. C-Index, C-Index/II, SpeedRead, ReadShare and PowerFail Protection are exclusive trademarks of Trio Systems LLC.


	\|CI2 Home\|Literature\|Support\|Documentation\|Y2K\|Order\|License\|