SynchroniCity - Data Model Mapper

The Data Model Mapper tool enables to convert several file types (e.g. CSV, Json, GeoJson) to the different Data Models defined in the SynchroniCity Project. The files in input can contain either rows, JSON objects or GeoJson Features, each of them representing an object to be mapped to an NGSI entity, according to the selected Data Model. The tool needs a file, the Mapping JSON, in order to know how to map each source field of the parsed row/object to the destination fields. In addition the tool writes the result either in a local file or directly to a configured Orion Context Broker.

Deploying the Data Model Mapper

The Data Model Mapper is a NodeJs CLI tool that performs its task and then exits. The tool requires NodeJS version >= 8.11 to be installed. Once you installed the NodeJs version for your OS, first of all, clone the repository:

git clone https://gitlab.com/synchronicity-iot/data-model-mapper.git

Configuration file

Go to the root folder of the cloned repository and rename the config.template.js to config.js. Change the following parts in the file.

  • writers, if you want to change default handlers for writing the resulting mapped NGSI entities to different outputs. Allowed values:
  writers: ["orionWriter", "fileWriter"]
  • If you keep the orionWriter, change following parts according to your configuration (e.g., replace the placeholders "<>" with the corresponding information):
  config.orionWriter = {
    orionUrl: "<my-orion-url>", 
    orionAuthHeaderName: "<my-authHeader>",
    orionAuthToken: "<my-authToken>",
    fiwareService: "<my-fiwareService>",
    fiwareServicePath: "<my-fiwareServicePath",
    enableProxy: false,
    proxy: '<my-proxy-url>',
    skipExisting: false,
    updateMode: "APPEND|REPLACE"
  }

More specifcially:

  • orionUrl: The Orion Context Broker endpoint (baseUrl without '/v2...') where mapped entities will be stored (e.g. http://localhost:1026)

  • orionAuthHeaderName: Authorization Header name (e.g. X-Auth-Token or Authorization) for Orion requests. Leave blank if any.

  • orionAuthToken: Authorization Token value for Orion request (e.g. Bearer XXX). Leave blank if any

  • fiwareService: Fiware-Service header to be put in the Orion request (e.g. RZservice)

  • fiwareServicePath: Fiware-ServicePath header to be put in the Orion request (e.g. /servicePath)

  • enableProxy: Enable requests if the tool is behind a Proxy

  • proxy: Proxy Url in the form http://user:pwd@proxyHost:proxyPort

  • skipExisting: Skip mapped entities (same ID) already existing in the Orion CB, otherwise update them according to updateMode parameter

  • updateMode: Possible values are APPEND or REPLACE. If to append or replace attributes in the existing entities.

  • If you keep the fileWriter change following parts according to your configuration (e.g., replace the placeholders "<>" with the corresponding information):

config.fileWriter = {
    filePath: "/path/to/result.json"
}

More specifically:

  • fileWriter: File path and name where mapped entities will be written.

Install and run

To install all the required Nodejs libraries, type the following command (in the root folder):

npm install

After choosing appopriate CLI parameters (see below), start it up with the command:

node mapper <cliParameters>

Change <cliParameters> with the followings:

  • -s or --sourceDataPath : The path (relative or absolute) of the source file. (e.g. "path/to/sourcefile.csv")
  • -m or --mapPath: The path (relative or absolute) of the JSON Map file, specifying the mapping between source fields and destrination fields. It is a JSON , where the key-value pairs represent the mapping for each row contained in the source file. (e.g. "path/to/map.json")
  • -d or --targetDataModel: The name of the target SynchroniCity Data Model

Following parameters are relative to fields that will compose the generated IDs of mapped entities, according to the SynchroniCity’s Entity ID Recommendation. The Entity Name (last part of ID pattern), is generated either automatically or by specifying it in the dedicated field entitySourceId of the JSON Map, as described in the Mapping Guide of the full documentation.

  • --si or --site: The site field of the generated ID can represent a RZ, City or area that includes several different IoT deployments, services or apps (e.g., Porto, Milano, Santander, Aarhus, Andorra …).
  • --se or --service: Represents a smart city service/application domain for example parking, garbage, environmental etc.
  • --gr or --group: The group part can be used for grouping assets under the same service and/or provider (so it can be used to identify different IoT providers).

Following parameters are relative to the rows range (start, end) of the input file that will be mapped. It is useful when you want to map only a part of the input file; If omitted, by default the tool will parse all the rows.

  • --rs or --rowStart: Row of the input file from which the mapper will start to map objects (Allowed values are integers >= 0).
  • --re or --rowEnd: Last Row of the input file that will be mapped (Allowed values are integers > 0 or Infinity value, indicating "until the end of file").

The following is an example of launching the tool:

node mapper -s "path/to/sourcefile.csv" -m "path/to/mapFile.json -d "WeatherObserved" --site Milan --service WeatherService --group provider1

IMPORTANT: If these CLI parameters are not provided in the node mapper command, the default ones, contained in the config.js file will be used. You can as well change these parameters in the file and avoid to use here when launching the command. See the full documentation here.

Mapping guide and examples

For further details about other specific configurations, as well as the Mapping Guide and examples about how to provide input files, see the full documentation in the GitLab repo

← Understanding DatamodelsAuthentication →