Developer Blog

20/02/2017 by Kvaser

Converting to plain ASCII

This is an update of an older blog post and is now taking advantage of the Python canlib package v1.7.

This is the second post in a 3-part series about using the Converter Library (kvlclib) in CANlib SDK:

Writer formats and properties (1 of 3)
Converting to plain ASCII (2 of 3)
Special converter conditions (3 of 3)

The first part of this blog series looked into converter formats and properties, we will now continue and convert events from a given log file.

Let us assume that we have a log file, mylog.kme50, that we would like to convert to Plain text (.txt) format. The log file was created either by using the Kvaser Memorator Config Tool, or by using the kvmlib as described in an earlier blog post.¹

Starting with the last code example we wrote in the previous blog article in this series, we now add setting up the name and format of our input file at the end.

# set up formats
out_fmt = kvlclib.WriterFormat(kvlclib.FileFormat.PLAIN_ASC)
in_fmt = kvlclib.ReaderFormat(kvlclib.FileFormat.KME50)

# set resulting output file name taking advantage of the extension
# defined in the format.
out_file = "myresult." + out_fmt.extension
print("Output filename is '%s'" % out_file)

# create converter
cnv = kvlclib.Converter(out_file, out_fmt)

# Set input file and format
cnv.setInputFile(INPUT_FILE, kvlclib.FileFormat.KME50)View sourceCopy to clipboard

Let us also add a loop that converts the events one by one until we reach the end of our input file. The last thing we do is to call Converter.flush() in order to flush output to disk.

# Convert events from input file one by one until EOF is reached
while True:
    try:
        cnv.convertEvent()
    except kvlclib.KvlcEndOfFile:
        break

# Force a flush so that the resulting file is written to disk.
cnv.flush()View sourceCopy to clipboard

Our first try at writing a kme50 converter program now looks as follows.

First try at writing a kme50 converter program

# 04_first_try_converting.py
from canlib import kvlclib


INPUT_FILE = 'mylog.kme50'

# set up formats
out_fmt = kvlclib.WriterFormat(kvlclib.FileFormat.PLAIN_ASC)
in_fmt = kvlclib.ReaderFormat(kvlclib.FileFormat.KME50)

# set resulting output file name taking advantage of the extension
# defined in the format.
out_file = "myresult." + out_fmt.extension
print("Output filename is '%s'" % out_file)

# create converter
cnv = kvlclib.Converter(out_file, out_fmt)

# Set input file and format
cnv.setInputFile(INPUT_FILE, kvlclib.FileFormat.KME50)

# Convert events from input file one by one until EOF is reached
while True:
    try:
        cnv.convertEvent()
    except kvlclib.KvlcEndOfFile:
        break

# Force a flush so that the resulting file is written to disk.
cnv.flush()View sourceCopy to clipboard

Running the above code, assuming everything works as expected, the file myresult.txt is created. But running the same code again results in an error:

canlib.kvlclib.exceptions.KvlcGeneralError: Output file already exists (-6)View sourceCopy to clipboard

Each converter has a number of properties that effect their inner workings. One of these properties is OVERWRITE which defaults to ‘0’, and that is the reason the code failed when running the second time – the resulting file already existed and the converter was not allowed to overwrite it.

As we saw in the previous blog post, we can ask if a property is supported by a specific format. So let us create a function that checks if a given property is supported. If the property is supported – and a value was supplied to the function – the function sets the property to the given value. We also take the opportunity to print the default value of the property to the console.

def trySetProperty(cnv, property, value=None):
    # Check if the format supports the given property
    if cnv.format.isPropertySupported(property):
        # If a value is specified, set the property to this value
        if value is not None:
            cnv.setProperty(property, value)

        # Get the property's default value
        default = cnv.format.getPropertyDefault(property)
        print('  %s is supported (Default: %s)' %
              (property, default))

        # Get the property's current value
        value = cnv.getProperty(property)
        print('    Current value: %s' % value)
    else:
        print('  %s is not supported' % property)View sourceCopy to clipboard

Now let us use the trySetProperty() function and set the property OVERWRITE to ‘1’, we also limit the output to only the first channel by setting the CHANNEL_MASK property. Another useful property for the Plain text format is WRITE_HEADER which adds a nice header to the output file, so let us set that property as well. The final property we set is LIMIT_DATA_BYTES which controls how many data bytes to print. In the ASCII format this directly affects how wide our data table will be, and since we do not have any CAN FD² messages in our input log, we set this property to 8 (bytes).

# allow output file to overwrite existing files
trySetProperty(cnv, kvlclib.Property.OVERWRITE, 1)

# we are only interested in the first channel
cnv.setProperty(kvlclib.Property.CHANNEL_MASK, 1)

# add nice header to the output file
trySetProperty(cnv, kvlclib.Property.WRITE_HEADER, 1)

# we are converting CAN traffic with max 8 bytes, so we can minimize
# the width of the data output to 8 bytes
trySetProperty(cnv, kvlclib.Property.LIMIT_DATA_BYTES, 8)View sourceCopy to clipboard

Collecting all our improvements so far, we end up with the following program.

A sample program to convert a KME50 log file into plain ASCII.

# 05_convert_kme.py
from canlib import kvlclib

def trySetProperty(cnv, property, value=None):
    # Check if the format supports the given property
    if cnv.format.isPropertySupported(property):
        # If a value is specified, set the property to this value
        if value is not None:
            cnv.setProperty(property, value)

        # Get the property's default value
        default = cnv.format.getPropertyDefault(property)
        print('  %s is supported (Default: %s)' %
              (property, default))

        # Get the property's current value
        value = cnv.getProperty(property)
        print('    Current value: %s' % value)
    else:
        print('  %s is not supported' % property)

# set up formats
out_fmt = kvlclib.WriterFormat(kvlclib.FileFormat.PLAIN_ASC)
in_fmt = kvlclib.ReaderFormat(kvlclib.FileFormat.KME50)

# set resulting output file name taking advantage of the extension
# defined in the format.
out_file = "myresult." + out_fmt.extension
print("Output filename is '%s'" % out_file)

# create converter
cnv = kvlclib.Converter(out_file, out_fmt)

# Set input file and format
cnv.setInputFile(INPUT_FILE, kvlclib.FileFormat.KME50)

# allow output file to overwrite existing files
trySetProperty(cnv, kvlclib.Property.OVERWRITE, 1)

# we are only interested in the first channel
cnv.setProperty(kvlclib.Property.CHANNEL_MASK, 1)

# add nice header to the output file
trySetProperty(cnv, kvlclib.Property.WRITE_HEADER, 1)

# we are converting CAN traffic with max 8 bytes, so we can minimize
# the width of the data output to 8 bytes
trySetProperty(cnv, kvlclib.Property.LIMIT_DATA_BYTES, 8)

# Convert events from input file one by one until EOF is reached
while True:
    try:
        cnv.convertEvent()
    except kvlclib.KvlcEndOfFile:
        break

# Force a flush so that the resulting file is written to disk.
cnv.flush()View sourceCopy to clipboard

Running the full program results in the following output.

                              Kvaser Memorator Log
                              ====================

Converted from Memorator Binary format at: 2018-08-14 07:47:10

                                                        
Settings:
   Format of data field: DEC
   Format of id field:   DEC
   Timestamp Offset:     0.000000 s
   CAN channel:          1

       Time Chan   Identifier Flags     DLC  Data                            Counter
====================================================================================
   0.242162  Trigger (type=0x1, active=0x00, pre-trigger=0, post-trigger=-1)
  32.003407  1         501    Rx         8   41  23   0   0   0   0   0   0        2
  32.006923  1         501    Rx         8  234   0   0   0   0   0   0   0        4
  32.007795  1         504    Rx         8   51 179  79  67   0   0   0   0        6
  32.008373  1         402    Rx         8   51  51 134  66   0   0   0   0        8
  32.009127  1         402    Rx         8   51  51 243  64   0   0   0   0       10
  32.009925  1         504    Rx         8    0   0  56  66   0   0   0   0       12
  32.010670  1         503    Rx         8    0   0  52  67   0   0   0   0       14
  32.011413  1         504    Rx         8  154  25  90  67   0   0   0   0       16
  32.012167  1         402    Rx         8  205 204 140  64   0   0   0   0       18
  32.015623  1         504    Rx         8  205 204  68  65   0   0   0   0       20View sourceCopy to clipboard

This ends the second blog article in which we created a simple program that used the Kvaser Converter library, kvlclib, to convert a kme50 log file into a plain text file. In the last article of this series, we will look at some special conditions that can arise when converting a log file.

If you have any questions, comments or suggestion for future blog articles, you can contact us directly via mail at support@kvaser.com.

Footnotes

1 Reading from a Kvaser Memorator and saving the logged data using kvmlib was done in www.kvaser.com/developer-blog/digging-deeper-into-kvmlib

2 CAN FD allows data fields up to 64 bytes per frame. See the blog article www.kvaser.com/developer-blog/can-fd for more information about CAN FD.

This article has been updated. To view the original, click on the box below.

Original Article

This blog uses now deprecated functions, see blog “Improved API in Python canlib v1.5” for more information.

This is the second post in a 3-part series about using the Converter Library (kvlclib) in CANlib SDK:

Writer formats and properties (1 of 3)
Converting to plain ASCII (2 of 3)
Special converter conditions (3 of 3)

The first part of this blog series looked into converter formats and properties, we will now continue and convert events from a given log file.

Let us assume that we have a log file, mylog.kme50, that we would like to convert to Plain text (.txt) format. The log file was created either by using the Kvaser Memorator Config Tool, or by using the kvmlib as described in an earlier blog post.^[1]

Starting with the latest code sample we wrote in the first part of this blog series, we now add some code for setting up the format and filename of our input file at the end.

import canlib.kvlclib as kvlc

# set output format
fmt = kvlc.WriterFormat(kvlc.FILE_FORMAT_PLAIN_ASC)
# the name of the formatter is fetched using kvlcGetWriterName() internally
print("Output format is '%s'" % fmt.name)

# set resulting output filename taking advantage of the extension defined in
# the format. (Uses kvlcGetWriterExtension() under the hood.)
outfile = "myresult." + fmt.extension
print("Output filename is '%s'" % outfile)

# create converter
kc = kvlc.Kvlclib(outfile, fmt)

# Set input filename and format
inputfile = "mylog.kme50"
print("Input filename is '%s'" % inputfile)
kc.setInputFile(inputfile, file_format=kvlc.FILE_FORMAT_KME50)
View sourceCopy to clipboard

Let us now add a loop that converts the events one by one until we reach the end of our input file.

# Convert events from input file one by one until EOF is reached
while True:
    try:
        kc.convertEvent()
    except kvlc.KvlcEndOfFile:
        breakView sourceCopy to clipboard

The last thing we need to do is to call kvlcDeleteConverter() in order to flush output to disk and free memory.

# Delete converter, flush result to disk and free memory
kc.deleteConverter()View sourceCopy to clipboard

Running the above code once, the file myresult.txt is created. But running the same code again results in an error:

  File "c:\dev\python\pycanlib\src\canlib\kvlclib.py", line 459, in convertEvent
    self.dll.kvlcConvertEvent(self.handle)
  File "c:\dev\python\pycanlib\src\canlib\kvlclib.py", line 408, in _kvlc_err_check
    raise KvlcError(self, result)
canlib.kvlclib.KvlcError: [KvlcError] convertEvent: Output file already exists (-6)View sourceCopy to clipboard

Each converter has a number of properties that effect their inner workings. One of these properties is KVLC_PROPERTY_OVERWRITE which defaults to ’0’, and that is the reason the code failed when running the second time – the resulting file already existed and the converter was not allowed to overwrite the existing file.

def trySetProperty(converter, property, value=None):
    # Check if the format supports the given property
    if converter.format.isPropertySupported(property):

        # If a value was specified, set the property to this value
        if value is not None:
            converter.setProperty(property, value)

        # get the property's default value
        default = converter.getPropertyDefault(property)
        print(" PROPERTY_%s is supported (Default: %s)" %
              (property['name'], default))

        # get the property's current value
        value = converter.getProperty(property)
        print("	Current value: %s" % value)
    else:
        print(" PROPERTY %s is not supported" %
              (property['name']))View sourceCopy to clipboard

Listing 4: Defining a function that tries to set a given property.

Now let us use the trySetProperty() function defined in Listing 4 and set the KVLC_PROPERTY_OVERWRITE to ‘1’. A second useful property for the Plain text format is KVLC_PROPERTY_WRITE_HEADER which adds a nice header to the output file so let us set that property as well. The final property we set is KVLC_PROPERTY_LIMIT_DATA_BYTES which controls how many data bytes to print. In the ASCII format this directly affects how wide our data table will be, and since we do not have any CAN FD^[2] messages in our input log, we set this property to 8 (bytes).

# allow output file to overwrite existing files
trySetProperty(kc, kvlc.PROPERTY_OVERWRITE, 1)

# add nice header to the output file
trySetProperty(kc, kvlc.PROPERTY_WRITE_HEADER, 1)

# we are converting CAN traffic with max 8 bytes, so we can minimize the width
# of the data output to 8 bytes
trySetProperty(kc, kvlc.PROPERTY_LIMIT_DATA_BYTES, 8)View sourceCopy to clipboard

Listing 5: Setting some convenient properties when using the plain text writer.

Collecting all our improvements so far, we end up with the program shown in Listing 6.

import canlib.kvlclib as kvlc


def trySetProperty(converter, property, value=None):
    # Check if the format supports the given property
    if converter.format.isPropertySupported(property):

        # If a value was specified, set the property to this value
        if value is not None:
            converter.setProperty(property, value)

        # get the property's default value
        default = converter.getPropertyDefault(property)
        print(" PROPERTY_%s is supported (Default: %s)" %
              (property['name'], default))

        # get the property's current value
        value = converter.getProperty(property)
        print("	Current value: %s" % value)
    else:
        print(" PROPERTY %s is not supported" %
              (property['name']))


# set output format
fmt = kvlc.WriterFormat(kvlc.FILE_FORMAT_PLAIN_ASC)
# the name of the formatter is fetched using kvlcGetWriterName() internally
print("Output format is '%s'" % fmt.name)

# set resulting output filename taking advantage of the extension defined in
# the format. (Uses kvlcGetWriterExtension() under the hood.)
outfile = "myresult." + fmt.extension
print("Output filename is '%s'" % outfile)

# create converter
kc = kvlc.Kvlclib(outfile, fmt)

# Set input filename and format
inputfile = "mylog.kme50"
print("Input filename is '%s'" % inputfile)
kc.setInputFile(inputfile, file_format=kvlc.FILE_FORMAT_KME50)

# allow output file to overwrite existing files
trySetProperty(kc, kvlc.PROPERTY_OVERWRITE, 1)

# add nice header to the output file
trySetProperty(kc, kvlc.PROPERTY_WRITE_HEADER, 1)

# we are converting CAN traffic with max 8 bytes, so we can minimize the width
# of the data output to 8 bytes
trySetProperty(kc, kvlc.PROPERTY_LIMIT_DATA_BYTES, 8)

# Convert events from input file one by one until EOF is reached
while True:
    try:
        kc.convertEvent()
    except kvlc.KvlcEndOfFile:
        break

# Delete converter, flush result to disk and free memory
kc.deleteConverter()View sourceCopy to clipboard

Listing 6: Simple program to convert a kme50 log file to plain text output.

Our second attempt as shown in Listing 6 can be run multiple times, and the top part of the generated file myresult.txt is shown in Listing 7.

                               Kvaser Memorator Log
                               ====================

 Converted from Memorator Binary format at: 1/11/2017 15:54:02


Settings:
    Format of data field: DEC
    Format of id field:   DEC
    Timestamp Offset:     0.000000000 s
    CAN channel:          1 2

        Time Chan  Identifier Flags        DLC   Data                                     Counter
=============================================================================================
 0.247909525 Trigger (type=0x1, active=0x00, pre-trigger=0, post-trigger=-1)
10.648166375 1          55    Rx            1   2                                           1
10.648166375 2          55    Rx            1   2                                           2
12.464742375 1         340    Rx            8   0   0   0   0   0   0   0   0               3
12.464742375 2         340    Rx            8   0   0   0   0   0   0   0   0               4View sourceCopy to clipboard

Listing 7: Top rows of the plain text output result from the conversion.

In this article, we created a simple program to convert a kme50 log file to plain text. In the last article of this series, we will look at the special conditions that can arise when converting a log file.

If you have any questions, comments or suggestion for future blog articles, you can contact us directly at support@kvaser.com.

Footnotes

[1] Reading from a Kvaser Memorator and saving the logged data using kvmlib was done in www.kvaser.com/developer-blog/digging-deeper-into-kvmlib

[2] CAN FD allows data fields up to 64 bytes per frame. See the blog article www.kvaser.com/developer-blog/can-fd for more information about CAN FD.

Magnus Carlsson

Magnus Carlsson is a Software Developer for Kvaser AB and has developed firmware and software for Kvaser products since 2007. He has also written a number of articles for Kvaser’s Developer Blog dealing with the popular Python language.

Developer Blog

Converting to plain ASCII

Footnotes

Original Article

Footnotes

Magnus Carlsson

Categories

RSS Copy Link

HQ Kvaser AB

Support

Company

Policies

Social

Developer Blog

Converting to plain ASCII

Footnotes

Original Article

Magnus Carlsson

Related posts

Kvaser NuGet Packages and Visual Studio

KVASER CANlib and Python Part 3: U100 & Python Highlights

Using CANlib Visual Studio 2017 C#.NET (canlibCLSNET)

Using CANlib Visual Studio 2017 VB canlibCLSNET

Using CANlib Visual Studio 2017 VB .NET Standard 2.0

Using CANlib Visual Studio 2017 C# .NET Standard 2.0

Categories

RSS Copy Link