==============================================================================
CYCLOS APPLICATION NOTE                             Bill Goodman - 1998 Sep 22

Hi-Note File Format
==============================================================================


PDB File Format
---------------
During HotSync, the backup conduit creates a file named "Hi-NoteDB.PDB" in the
user's "Backup" directory. This file contains a complete copy of the Hi-Note
database on the PalmPilot. Unfortunately, the format of "PDB" files is not
well documented (see Appendix A). As a result, the following information is
incomplete and may contain errors.

A generic PDB file has the following layout:

    PDBHeader
    recordCnt x (PDBDataRecord | PDBRsrcRecord)
    padding (to 4-byte boundary?)
    AppInfo block [optional]
    SortInfo block [optional]
    recordCnt x (record data block)

The database structures are defined below. All numeric values are written in
big endian format.

// File header
typedef struct
    {
    char name[32];                  // DB name
    unsigned short flags;           // Flags:
        #define pdbFlagOpen             0x8000
        #define pdbFlagBackup           0x0008
        #define pdbFlagAppInfoDirty     0x0004
        #define pdbFlagReadOnly         0x0002
        #define pdbFlagResource         0x0001
    unsigned short version;         // Version:
        #define pdbVerShowSecret        0x0010
        #define pdbVerExclusive         0x0008
        #define pdbVerLeaveOpen         0x0004
        #define pdbVerReadOnly          0x0001
        #define pdbVerWrite             0x0002
        #define pdbVerReadWrite         0x0003
    unsigned long creationDate;     // DB creation date
    unsigned long modificationDate; // DB modification date
    unsigned long backupDate;       // DB backup date
    unsigned long modificationCnt;
    unsigned long appInfoOffset;    // File offset of AppInfo block
    unsigned long sortInfoOffset;   // File offset of SortInfo block
    unsigned long type;             // DB type code
    unsigned long creator;          // DB creator code
    unsigned long uniqueID;
    unsigned long nextRecordID;
    unsigned short recordCnt;       // Number of records in DB
    } PDBHeader;

// Data record index entry
typedef struct
    {
    unsigned long offset;           // File offset of record data
    unsigned char attr;             // Record attribute
        #define dmRecAttrDelete         0x80    // True if record should be
deleted next sync
        #define dmRecAttrDirty          0x40    // True if record should be
archived next sync
        #define dmRecAttrBusy           0x20    // True if record is in use
        #define dmRecAttrSecret         0x10    // True if record is
password protected
        #define dmRecAttrCategoryMask   0x0F    // Mask for category
    unsigned char uniqIDH;          // Unique ID (high byte)
    unsigned char uniqIDM;          // Unique ID (middle byte)
    unsigned char uniqIDL;          // Unique ID (low byte)
    } PDBDataRec;

// Resource record index entry
typedef struct
    {
    unsigned long resType;          // Resource type code
    unsigned short resID;           // Resource ID;
    unsigned long offset;           // File offset of record data
    } PDBRsrcRec;

// AppInfo block (some applications append additional data to this block)
#define dmRecNumCategories  16      // Number of categories in standard
AppInfo block
#define dmCategoryLength    16      // Length of category name (15 chars + 0)
typedef struct
    {
    unsigned short renamedCategories;
    char categoryLabels[dmRecNumCategories][dmCategoryLength];
    unsigned char categoryUniqIDs[dmRecNumCategories];
    unsigned char lastUniqID;
    } AppInfoBlk;


Hi-Note Database Definition
---------------------------
The Hi-Note database is named "Hi-NoteDB". It has a creator code of 'HNA1' and
a type code of 'HND2'. Interface code should always check the type code since
this only changes when the internal format of the database changes (a type
code of 'HND1' indicates a database created by Hi-Note 1.00 which is not
described in this document).

Hi-Note uses an extended version of the AppInfo block to store various
preference information. This information may vary from one version of Hi-Note
to another, so interface code should not assume any set length. Data in this
block should not be modified.

When creating a new PDB file, a default AppInfo block may be constructed by
creating an AppInfo block consisting of the standard fields plus 49 additional
bytes containing the Hi-Note preference data. After clearing the block to
zeroes, the fields should be initialized to the following values:

    renamedCategories = 0x0006
    categoryLabels[0][] = "Unfiled"
    categoryLabels[1][] = "Business"
    categoryLabels[2][] = "Personal"
    categoryUniqIDs[0] = 0
    categoryUniqIDs[1] = 1
    categoryUniqIDs[2] = 2
    categoryUniqIDs[3] = 3
    categoryUniqIDs[4] = 4
    categoryUniqIDs[5] = 5
    categoryUniqIDs[6] = 6
    categoryUniqIDs[7] = 7
    categoryUniqIDs[8] = 8
    categoryUniqIDs[9] = 9
    categoryUniqIDs[10] = 10
    categoryUniqIDs[11] = 11
    categoryUniqIDs[12] = 12
    categoryUniqIDs[13] = 13
    categoryUniqIDs[14] = 14
    categoryUniqIDs[15] = 15
    lastUniqID = 15

    HN preference data =
        00
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        04 4C 01 00 00 FF 00 00 00 00 00 00 00 00 00 00
        00 00 00 00 00 00 00 00 00 00 FF FF FF FF 00 00

Hi-Note uses one data record for each note. Notes may contain either text or a
picture. The note hierarchy is specified by a combination of the record index,
which indicates the note's position in the database, and the "level" field,
which indicates the note's hierarchical level. If the level of a note is the
same as the preceding note, it is a sibling of the preceding note. If the
level of a note is higher than the preceding note, it is  a child of the
preceding note. If the level of a note is lower than the preceding note, then
it is the sibling of an earlier note in the hierarchy. As an example, the
notes in the following list are indented to show their hierarchical
relationship. The "level" field is listed in parentheses:

    Note 1.0.0          (0)
    Note 2.0.0          (0)
        Note 2.1.0      (1)
        Note 2.2.0      (1)
            Note 2.2.1  (2)
            Note 2.2.2  (2)
        Note 2.3.0      (1)
            Note 2.3.1  (2)
    Note 3.0.0          (0)

The category of a note is assigned by setting the "dmRecAttrCategoryMask"
field in the record attribute to a value between 0 and 15. All notes must be
assigned to the same category as their parent. Only root-level notes may be
assigned to different categories.

Notes may be marked private or secret by setting the "dmRecAttrSecret" flag in
the record attribute. If a note is marked secret, all children of the note
must be marked secret, too.


Note Record Definition
----------------------
The structure of a note record is specified below.

// Note record
typedef struct
    {
    Byte flags;                     // Item flags
        #define flagsTwistOpen          0x80    // True if twistdown is in
expanded position
        #define flagsDisplayable        0x40    // True if note is displayable
        #define flagsPict               0x20    // True if note is a picture
        #define flagsReadonly           0x10    // True if note is read-only
    Byte level;                      // Hierarchy level
    char text[n];                    // Note text (0 terminated)
    PictData pictureData;            // Picture data (picture notes only)
    } NoteRec;

The "flagsDisplayable" flag is true if the record would be visible based on
the settings of the twistdown controls of the other notes in the database. No
consideration of the category or secret status of the record is used in
determining displayability.

The "level" field indicates the indentation level of the note. Its value is 0
for root-level notes. The maximum value for the level of a note is one greater
than the level of the preceding note. Since Hi-Note supports 10 levels of
hierarchy, no note may have a value greater than 9.

The "text" field contains the main text for a text note or the title for a
picture note. For text notes, the maximum size is 28,672. For picture notes,
the maximum size is 70.


Picture Definition
------------------
Only notes which contain pictures include the "pictureData" field. The picture
data structure is described below.

// Picture data
typedef struct
    {
    Word width;         // Width of picture in pixels
    Word height;        // Height of picture in pixels
    Byte pixelData[n];  // Pixel data
    } PictData;

Valid values for the "width" field are: 160, 320, 480, 640 and 800.

Valid values for the "height" field are: 125, 250, 375 and 500.

"pixelData" consists of black-and-white pixel data in a compressed format.
Pixels are specified one row at a time running from left to right. Pixel rows
are specified from top to bottom. The compression algorithm is a simple
duplicate removal algorithm which reduces runs of 4 or more duplicates. Data
is compressed into blocks consisting of a 16-bit compression code followed by
1 to 512 bytes of data. The compression code is defined as:

    DDDDDDCCCCCCCCCC: Compression code
          CCCCCCCCCC: Copy count (0-511) - one less than the number of data
                      bytes following the compression code.
              DDDDDD: Duplicate count (0-63) - number of times to repeat the
                      last byte of the data string

The following snippet uses the compression algorithm to compress some source
picture data.

unsigned char *srcPtr;  // Pointer to start of source data
unsigned long srcCnt;   // Length of source data
unsigned char wrBuf[514];
unsigned char *wrPtr;
unsigned long wrCnt;
unsigned long dupCnt;
unsigned char lastByte;

wrPtr = &wrBuf[2];
wrCnt = 0;
while (srcCnt)
    {
    lastByte = *srcPtr++;
    *wrPtr++ = lastByte;
    srcCnt--;
    wrCnt++;

    // Check for write trigger conditions
    if (srcCnt > 1)
        if ((lastByte == srcPtr[0]) && (lastByte == srcPtr[1]))
            goto DoWrite;   // 3 bytes match (write when 3 match even though
                            // there is no reduction since this minimizes
                            // number of writes due to exceeding the maximum
                            // non-repeat length
    if (wrCnt == 512)
        goto DoWrite;   // Maximum non-repeat length?
    if (!srcCnt)
        goto DoWrite;   // End of source data
    continue;

    // Write required - look for up to 63 duplicates of the last byte in
the buffer
DoWrite:
    dupCnt = 0;
    while (srcCnt)
        {
        if (*srcPtr != lastByte)
            break;  // Next byte is not a duplicate
        srcPtr++;
        srcCnt--;

        if (++dupCnt == 63)
            break;  // Maximum duplicate count
        }
    wrCnt--;
    wrBuf[0] = (dupCnt << 2) | (wrCnt >> 8);
    wrBuf[1] = wrCnt;
    CopyToOutput(wrBuf, wrCnt + 3);
    wrPtr = &wrBuf[2];
    wrCnt = 0;
    }


------------------------------------------------------------------------------
APPENDIX A: NOTES ON PDB FORMAT (From the 3Com FAQ)
------------------------------------------------------------------------------

Question:
    How can I find out about the *.pdb file formats?

Answer:
    At present we do not document the .PDB format or the Desktop data file
(*.dat, *.dba, *.tda) formats. Some third-party developers have  experimented
with deciphering the format. Search some of the sites listed in the "Other
Sites" section of Roadcoders for a utility program such as File Dump 1.0 by
Pat Beirne.

    We recognize the importance for some developers to have these formats
available, but we do not currently have an officially documented data format.
Also note that these formats have a very good chance of changing in future
revisions of the PalmOS and supporting applications, even if they appear to
work presently.

    The Desktop data file formats are actually the result of serialized MFC
objects, even the files on the Macintosh. Regretfully, we do not know their
precise binary format; they are in an undisclosed format proprietary to
Microsoft. An investigation showed that they contain some clear text, and
remnants of deleted, but not yet removed records.

    It is possible to write an independent MSVC++ 4.1 MFC application to read
the (*.dat) files; note that they are MFC version dependent. The *.dba and
*.tda files can be renamed to *.dat; the reason their names are different is
to identify what they are: dba (datebook archive), tda (todo archive).

    Oftentimes the best solution would be to write a conduit that moves data
directly from the PalmPilot to your end application. Performing all processing
in a single step is easier on the user, and more compatible with the PalmPilot
approach. You are welcome to use the conduit source code in the Conduit SDK to
get you on your way.



