DataReader

DataReader is a plugin that allows for reading feeds of information from several different types of sources of information. The plugin supports reading Excel files, databases using SQL, feeds using XML and JSON. The plugin also supports different types of outputs.

The plugin shows different arguments depending on the inputs and the outputs used:

Functionality

All of the different sources of information behave the same way: they all search for records, and translate the records fields to a certain format according to the requested output type. In the case of DataPool output types the plugin converts each incoming record to a DataPool command and executes it. In the case of Shared Memory the plugin creates a character separated string of fields and pushes to the shared memory.

Input Type - Excel

In this mode the plugin reads a sheet from an Excel file. The parameters in this mode are:

File Name: name of the file to be read.
Table/Sheet: name of the sheet to be read from within the file.
Key: this is the name of the column in the Excel file from which the keys to the DataTable ought to be taken.
From row: The row number in the Excel sheet from where to start reading the records.
Number of rows: The maximum number of records to read, starting from the row stated in “From row”.
Fields to Read: Specifies by name the columns to read.
DP Field Name: Specifies the field name in DataPool to where the output will be written.
Field is a: Specifies the type of the target.
Post Update Action: Specifies commands (either DataPool or Viz commands) to be executed right after the data is read and sent to the target.
Show Data: If “on” then it dumps the data the plugin reads to a DataPool variable called READER_CONSOLE. This is extremely useful for development and debugging purposes. This includes error messages too.
Return Variables Prefix: Specifies a prefix to add to the name of the DataPool variables that the plugin uses. For example, if the prefix is “STV_”, then the plugin will dump the debug information to STV_READER_CONSOLE.
Load Automatically: Reads the information automatically every certain amount of time. This amount of time is defined in “Automatic Load Period (in seconds)”.
Automatic Load Period (in seconds): Specifies the period of time between consecutive automatic reads.
Load: A button that invokes the action of reading the data.

Example

The following is a typical example of a table in an Excel sheet we would be interested in reading from using DataReader.

In this case, if the parameters are configured as:

File name = C:\temp\Indexes.xlsx
Table/Sheet = Indexes$
Key = Stocks
From Row = 5
Number of Rows = 3
Fields To Read = Stocks, Prices
DP Field Name = Indexes
Data Field = DataTable

then the plugin will dump to DataPool the following command:

Indexes[EURO STOXX 50 ]={
   {Stocks=""EURO STOXX 50 "";Prices=""3131.39"";Volumes=""-37.84 (-1.19%)"";};};
Indexes[CAC 40 ]={
   {Stocks=""CAC 40 "";Prices=""4265.04"";Volumes=""-47.26 (-1.10%)"";};};
Indexes[S&P TSX ]={
   {Stocks=""S&P TSX "";Prices=""15476.77"";Volumes=""-48.05 (-0.31%)"";};}; 
NUM_RECORDS=3;

Note that the “Key” parameter is used to obtain the column from which the indexes of the table.

In case the Data Field is of type DataArray:

Indexes[0]={
   {Stocks=""EURO STOXX 50 "";Prices=""3131.39"";Volumes=""-37.84 (-1.19%)"";};};
Indexes[1]={
   {Stocks=""CAC 40 "";Prices=""4265.04"";Volumes=""-47.26 (-1.10%)"";};}; 
Indexes[2]={
   {Stocks=""S&P TSX "";Prices=""15476.77"";Volumes=""-48.05 (-0.31%)"";};}; 
NUM_RECORDS=3;

In case the Data Field is of type DataStructure, the “Key” parameter is used as the name of the column from which to take the names of the DP field:

EURO STOXX 50={Stocks=""EURO STOXX 50 "";Prices=""3131.39"";Volumes=""-37.84 (-1.19%)"";};
CAC 40={Stocks=""CAC 40 "";Prices=""4265.04"";Volumes=""-47.26 (-1.10%)"";};
S&P TSX={Stocks=""S&P TSX "";Prices=""15476.77"";Volumes=""-48.05 (-0.31%)"";}; 
NUM_RECORDS=3;

In case the DataField is of type DataFieldArray, then the output will look like:

Indexes[0-8]=””EURO STOXX 50 "",""3131.39"",""-37.84 (-1.19%)"",""CAC 40 ",""4265.04"",""-47.26 (-1.10%)"",""S&P TSX "",""15476.77"",""-48.05 (-0.31%)""; 
NUM_RECORDS=9;

If the user chose to output the data to Shared Memory then the following value will be dump into the selected variable:
```
EURO STOXX 50|3131.39|-37.84 (-1.19%)|CAC 40|4265.04|-47.26 (-1.10%)|S&P TSX|15476.77|-48.05 (-0.31%)
```

The output is a string result of the concatenation of all the values. The separator is by default a pipe character (|), but it can be changed.

Input Type - SQL

In this mode the plugin reads a sheet from a database that supports SQL.

The parameters in this mode are:

Connection String: The connection string to connect to the database. Each type of database has a different connection string. A good reference on connection strings for several different databases can be found in http://www.connectionstrings.com/.
SQL Query: SQL command for querying the database.
Key: this is the name of the column in the read query or table from which the keys to the DataTable ought to be taken.
From row: The row number in the query/table from where to start reading the records.
Number of rows: The maximum number of records to read, starting from the row stated in “From row”.
Fields to Read: Specifies by name the columns to read.
DP Field Name: Specifies the field name in DataPool to where the output will be written.
Field is a: Specifies the type of the target.
Post Update Action: Specifies commands (either DataPool or Viz commands) to be executed right after the data is read and sent to the target.
Show Data: If “on” then it dumps the data the plugin reads to a DataPool variable called READER_CONSOLE. This is extremely useful for development and debugging purposes. This includes error messages too.
Return Variables Prefix: Specifies a prefix to add to the name of the DataPool variables that the plugin uses. For example, if the prefix is “STV_”, then the plugin will dump the debug information to STV_READER_CONSOLE.
Load Automatically: Reads the information automatically every certain amount of time. This amount of time is defined in “Automatic Load Period (in seconds)”.
Automatic Load Period (in seconds): Specifies the period of time between consecutive automatic reads.
Load: It’s a button that invokes the action of reading the data.

The different types of outputs are exactly as in the case of Excel.

Input Type - XML

This mode allows for reading XML feeds. These can be read from a remote server or from a local file.

These can be read from a remote server or from a local file.

The parameters in this mode are:

File Name: It can be either the full name of a file or a URI to a remote file to be read.
Use Authentication: Allows entering a user name and password In the case the “File Name” is a URI and the remote server requires Basic Authentication.
Avoid Cache: In the case of a remote file, avoid using cache mechanisms that avoid the refresh of data changing in the server side. This is useful when reading information that changes. The caching mechanism avoids the plugin to get the updates. When “Avoid Cache” is on the plugin gets the freshest information.
XPath: This is the expression that states what records to search for. For an explanation of the XPath syntax please refer to http://www.w3.org/TR/xpath/. For examples of how to use XPath please refer to http://msdn.microsoft.com/en-us/library/ms256086(v=vs.110).aspx.
Namespaces: Specifies namespaces for use in XPath expressions when it is necessary to define new namespaces externally. Namespaces are defined in the XML style, as a space-separated list of namespace declaration attributes. You can use this property to set the default namespace as well. An example of the definition of namespaces could be

xmlns:na='http://myserver.com' xmlns:nb='http://yourserver.com'
Key: this is the name of the field in the read records from which the keys to the DataTable ought to be taken.
From row: The row number in the read record from where to start reading the records.
Number of rows: The maximum number of records to read, starting from the row stated in “From row”.
Fields to Read: Specifies by name the fields to read from the records.
DP Field Name: Specifies the field name in DataPool to where the output will be written.
Field is a: Specifies the type of the target.
Post Update Action: Specifies commands (either DataPool or Viz commands) to be executed right after the data is read and sent to the target.
Show Data: If “on” then it dumps the data the plugin reads to a DataPool variable called READER_CONSOLE. This is extremely useful for development and debugging purposes. This includes error messages too.
Return Variables Prefix: Specifies a prefix to add to the name of the DataPool variables that the plugin uses. For example, if the prefix is “STV_”, then the plugin will dump the debug information to STV_READER_CONSOLE.
Load Automatically: Reads the information automatically every certain amount of time. This amount of time is defined in “Automatic Load Period (in seconds)”.
Automatic Load Period (in seconds): Specifies the period of time between consecutive automatic reads.
Load: A button that invokes the action of reading the data.

Input Type - JSON

This mode allows for reading JSON feeds. These can be read from a remote server or from a local file.

The parameters in this mode are:

File Name: It can be either the full name of a file or a URI to a remote file to be read.
XPath: This is the expression that states what records to search for. This is a subset of the standard XPath.
Key: this is the name of the field in the read records from which the keys to the DataTable ought to be taken.
From row: The row number in the read record from where to start reading the records.
Number of rows: The maximum number of records to read, starting from the row stated in “From row”.
Fields to Read: Specifies by name the fields to read from the records.
DP Field Name: Specifies the field name in DataPool to where the output will be written.
Field is a: Specifies the type of the target.
Post Update Action: Specifies commands (either DataPool or Viz commands) to be executed right after the data is read and sent to the target.
Show Data: If “on” then it dumps the data the plugin reads to a DataPool variable called READER_CONSOLE. This is extremely useful for development and debugging purposes. This includes error messages too.
Return Variables Prefix: Specifies a prefix to add to the name of the DataPool variables that the plugin uses. For example, if the prefix is “STV_”, then the plugin will dump the debug information to STV_READER_CONSOLE.
Load Automatically: Reads the information automatically every certain amount of time. This amount of time is defined in “Automatic Load Period (in seconds)”.
Automatic Load Period (in seconds): Specifies the period of time between consecutive automatic reads.
Load: It’s a button that invokes the action of reading the data.