Magnus Carlsson

Special converter conditions

This is the final post in a 3-part series about using the Converter Library (kvlclib) in CANlib SDK:

  1. Writer formats and properties (1 of 3)
  2. Converting to plain ASCII (2 of 3)
  3. Special converter conditions (3 of 3)

In the second part of this blog series, we created a simple program to convert a kme50 log file to plain text. We will now continue and look at the three special conditions that can arise when converting a log file; overrun, data truncation, and change in output filename:

Overrun can occur during logging with a Memorator if the bus load exceeds the logging capacity. This condition can be detected by calling kvlcIsOverrunActive(). Once the overrun has been noted, we need to manually reset the state using kvlcResetOverrunActive().

Data truncation occurs when the selected output converter can’t write all bytes in a data frame to the output file. This can happen if CAN FD data is extracted to a format that only supports up to 8 data bytes, e.g. KVLC_FILE_FORMAT_KME40. This condition is reset by a call to kvlcResetStatusTruncated().

New output filename is a change in the output filename occurs when the converter has created a new file, which can be the result when splitting output files (which we will look into briefly later).[1] The call to kvlcIsOutputFilenameNew() will return true when the last converted event resulted in the creation of a new output file. This condition will automatically reset during the next call to kvlcConvertEvent(). When a split has occurred, we can get the new output filename by calling kvlcGetOutputFilename().

Let us now create a function that, beside of doing the actual conversion, also looks for the above conditions and prints some informational messages to the user.

def convertEvents(kc):
    # Get estimated number of remaining events in the input file. This can be
    # useful for displaying progress during conversion.
    total = kc.eventCount()
    print("Converting about %d events..." % total)
    while True:
        try:
            # Convert events from input file one by one until EOF is reached
            kc.convertEvent()
            if kc.IsOutputFilenameNew():
                print("New output filename: %s" % kc.getOutputFilename())
                print("About %d events left to convert..." % kc.eventCount())
        except kvlc.KvlcEndOfFile:
            if kc.IsOverrunActive():
                print("NOTE! The extracted data contained overrun.")
                kc.resetOverrunActive()
            if kc.IsDataTruncated():
                print("NOTE! The extracted data was truncated.")
                kc.resetStatusTruncated()
            break

Listing 8: Defining a function that does the conversion and also check for special conditions.

import canlib.kvlclib as kvlc


def trySetProperty(converter, property, value=None):
    # Check if the format supports the given property
    if converter.format.isPropertySupported(property):

        # If a value was specified, set the property to this value
        if value is not None:
            converter.setProperty(property, value)

        # get the property's default value
        default = converter.getPropertyDefault(property)
        print(" PROPERTY_%s is supported (Default: %s)" %
              (property['name'], default))

        # get the property's current value
        value = converter.getProperty(property)
        print("	Current value: %s" % value)
    else:
        print(" PROPERTY %s is not supported" %
              (property['name']))


def convertEvents(kc):
    # Get estimated number of remaining events in the input file. This can be
    # useful for displaying progress during conversion.
    total = kc.eventCount()
    print("Converting about %d events..." % total)
    while True:
        try:
            # Convert events from input file one by one until EOF is reached
            kc.convertEvent()
            if kc.IsOutputFilenameNew():
                print("New output filename: %s" % kc.getOutputFilename())
                print("About %d events left to convert..." % kc.eventCount())
        except kvlc.KvlcEndOfFile:
            if kc.IsOverrunActive():
                print("NOTE! The extracted data contained overrun.")
                kc.resetOverrunActive()
            if kc.IsDataTruncated():
                print("NOTE! The extracted data was truncated.")
                kc.resetStatusTruncated()
            break


# set output format
fmt = kvlc.WriterFormat(kvlc.FILE_FORMAT_PLAIN_ASC)
# the name of the formatter is fetched using kvlcGetWriterName() internally
print("Output format is '%s'" % fmt.name)

# set resulting output filename taking advantage of the extension defined in
# the format. (Uses kvlcGetWriterExtension() under the hood.)
outfile = "myresult." + fmt.extension
print("Output filename is '%s'" % outfile)

# create converter
kc = kvlc.Kvlclib(outfile, fmt)

# Set input filename and format
inputfile = "mylog.kme50"
print("Input filename is '%s'" % inputfile)
kc.setInputFile(inputfile, file_format=kvlc.FILE_FORMAT_KME50)

# allow output file to overwrite existing files
trySetProperty(kc, kvlc.PROPERTY_OVERWRITE, 1)

# add nice header to the output file
trySetProperty(kc, kvlc.PROPERTY_WRITE_HEADER, 1)

# we are converting CAN traffic with max 8 bytes, so we can minimize the width
# of the data output to 8 bytes
trySetProperty(kc, kvlc.PROPERTY_LIMIT_DATA_BYTES, 8)

convertEvents(kc)

# Delete converter, flush result to disk and free memory
kc.deleteConverter()

Listing 9: Simple program to convert a kme50 log file to plain text output.

Running the program in Listing 9 now gives the following output:

Output format is 'Plain text'
Output filename is 'myresult.txt'
Input filename is 'mylog.kme50'
  PROPERTY_OVERWRITE is supported (Default: 0)
    Current value: 1
  PROPERTY_WRITE_HEADER is supported (Default: 0)
    Current value: 1
  PROPERTY_LIMIT_DATA_BYTES is supported (Default: 64)
    Current value: 8
Converting about 71253 events...
New output filename: myresult.txt
About 71252 events left to convert...
NOTE! The extracted data contained overrun.

As can be seen in the output, our input file contained over 7000 events and as a result the output file created was over 4MB. Let us now split the output files by also setting the property KVLC_PROPERTY_SIZE_LIMIT.

# split output files into max 2 MB files
trySetProperty(kc, kvlc.PROPERTY_SIZE_LIMIT, 2)

The output from our final run is shown below, we now got three output files, where the first two are about 2MB.

Output format is ’Plain text’
Output filename is ’myresult.txt’
Input filename is ’mylog.kme50’
  PROPERTY_OVERWRITE is supported (Default: 0)
    Current value: 1
  PROPERTY_WRITE_HEADER is supported (Default: 0)
    Current value: 1
  PROPERTY_LIMIT_DATA_BYTES is supported (Default: 64)
    Current value: 8
  PROPERTY_SIZE_LIMIT is supported (Default: 0)
    Current value: 2
Converting about 71253 events...
New output filename: myresult-part0.txt
About 71252 events left to convert...
New output filename: myresult-part1.txt
About 41473 events left to convert...
New output filename: myresult-part2.txt
About 11695 events left to convert...
NOTE! The extracted data contained overrun.

Footnotes

[1] Splitting of output files is controlled by the writer properties KVLC_PROPERTY_SIZE_LIMIT and KVLC_PROPERTY_TIME_LIMIT.

Feedback