Wiki source code of DataLoader

Last modified by Michael Sibayan on 2022/05/03 16:48

version	line-number	content
1.1	1	{{box cssClass="floatinginfobox" title="Table of contents"}}
	2	{{toc/}}
	3	{{/box}}
	4
	5	= Introduction =
	6
	7	The purpose of this application is to extract data from properly formatted ASCII text, human readable, tab delimited files and upload the data to a database. For the LEO project we have chosen to use Oracle as our database.
	8
	9	This application is written entirely in Java (jre1.8.0_25) and requires Oracle Database JDBC Driver 12c (ojdbc7).
	10
	11	The application and dependencies are packed into Loader.jar which can be run from any computer with jre 1.8 or higher installed using the command:
	12	{{code}}java -jar Loader.jar{{/code}}
	13	The main method does not take any parameters and should be run with an account with sufficient write privileges to all folders accessed by the application.
	14
	15	= Configuration File =
	16
	17	This application requires the configuration file “Dataloader.config” to exist in the application directory and be properly formatted. The format is a java.util.Properties “Configuration File”. Each configuration item is first defined by name (e.g. “configoption1”) and is equal to some value:
	18
	19	> Configoption1 = “the value of config option 1”
	20
	21	Sections, used solely for reference, are displayed in brackets [section1]. A sample configuration file is shown below:
	22
	23	> {{code language="<Apache"}}[General]
	24	schemas = leo_mini,leo_general,leo_east
	25	sources = c:\\temp\\in1,c:\\temp\\in2,c:\\temp\\in3
	26	output = c:\\temp\\out
	27
	28	[Database]
	29	user = mydatabaseusername
	30	password = mydatabasepassword
	31	tns = jdbc:oracle:thin:@mydatabase.arizona.edu:1521:dbsid
	32
	33	[Logging]
	34	log = e:\\web
	35	web = e:\\web{{/code}}
	36
	37	\|schemas\|Comma separated list of schemas to use (each must exist within the database and must be accessible from the log-on account)
	38	\|sources\|Comma separated list of folders containing measurement files (windows paths require the double '
	39	' since the string interprets the first backslash as an escape character
	40	\|output\|The path for output/processed files
	41	\|user\|Database user account for accessing the database (and schemas)
	42	\|password\|Database password for the user account
	43	\|tns\|TNS path for accessing the database
	44	\|log\|Path to write the log file
	45	\|web\|Path to write the files heartbeat.xml and sensorlist.xml
	46
	47	= Application Dependencies =
	48
	49	This application requires the Java runtime environment (JRE1.8 or greater) be installed on the host machine.
	50
	51	The following SQL queries are executed:
	52
	53	> Select SENSORID(INT), SENSORCODE(INT), SENSORTYPE(STR), METHODDESCRIPTION(STR), STATUS(CHR), LOGGERNAME(STR), MODULE(STR), CHANNEL(STR), ADDRESS(STR) from VIEW_SENSORS
	54
	55	> Select CALIBRATIONPARAMETERID(INT), PARAMETERCODE(STR), PARAMETERVALUE(DBL) from CALIBRATIONPARAMETERS where sensorid = XXXX
	56	>>This query is executed for each sensor
	57
	58	> Select SENSORTYPE(STR), VARIABLEINDEX(INT), INVARIABLEID(INT), VARIABLENAME(STR), VARIABLEUNIT(STR), NODATAVALUE(DBL), VALIDRANGEFROM(DBL), VALIDRANGETO(DBL) from VIEW_INPUT_VARIABLES_METADATA
	59
	60	> Insert into DATAVALUES (datavalue, localdatetime, sensorid, variableid) values (AAAA, to_date(‘BBBB/BB/BB BB:BB ', 'YYYY/MM/DD HH24:MI'), CC, DD)
	61	>> ~~ Where A, B, C, D are the actual values required ~~
	62
	63
	64
	65	= Measurement file format =
	66
	67	(!)This application will only process properly formatted JSON files.
	68
	69	{{include document="MeasurementFile"/}}