Wiki source code of DataLoader

Last modified by Michael Sibayan on 2022/05/03 16:48

author	version	line-number	content
		1	{{box cssClass="floatinginfobox" title="Table of contents"}}
		2	{{toc/}}
		3	{{/box}}
		4
		5	= Introduction =
		6
		7	The purpose of this application is to extract data from properly formatted ASCII text, human readable, tab delimited files and upload the data to a database. For the LEO project we have chosen to use Oracle as our database.
		8
		9	This application is written entirely in Java (jre1.8.0_25) and requires Oracle Database JDBC Driver 12c (ojdbc7).
		10
		11	The application and dependencies are packed into Loader.jar which can be run from any computer with jre 1.8 or higher installed using the command:
		12	{{code}}java -jar Loader.jar{{/code}}
		13	The main method does not take any parameters and should be run with an account with sufficient write privileges to all folders accessed by the application.
		14
		15	= Configuration File =
		16
		17	This application requires the configuration file “Dataloader.config” to exist in the application directory and be properly formatted. The format is a java.util.Properties “Configuration File”. Each configuration item is first defined by name (e.g. “configoption1”) and is equal to some value:
		18
		19	> Configoption1 = “the value of config option 1”
		20
		21	Sections, used solely for reference, are displayed in brackets [section1]. A sample configuration file is shown below:
		22
		23	> {{code language="<Apache"}}[General]
		24	schemas = leo_mini,leo_general,leo_east
		25	sources = c:\\temp\\in1,c:\\temp\\in2,c:\\temp\\in3
		26	output = c:\\temp\\out
		27
		28	[Database]
		29	user = mydatabaseusername
		30	password = mydatabasepassword
		31	tns = jdbc:oracle:thin:@mydatabase.arizona.edu:1521:dbsid
		32
		33	[Logging]
		34	log = e:\\web
		35	web = e:\\web{{/code}}
		36
		37	\|schemas\|Comma separated list of schemas to use (each must exist within the database and must be accessible from the log-on account)
		38	\|sources\|Comma separated list of folders containing measurement files (windows paths require the double '
		39	' since the string interprets the first backslash as an escape character
		40	\|output\|The path for output/processed files
		41	\|user\|Database user account for accessing the database (and schemas)
		42	\|password\|Database password for the user account
		43	\|tns\|TNS path for accessing the database
		44	\|log\|Path to write the log file
		45	\|web\|Path to write the files heartbeat.xml and sensorlist.xml
		46
		47	= Application Dependencies =
		48
		49	This application requires the Java runtime environment (JRE1.8 or greater) be installed on the host machine.
		50
		51	The following SQL queries are executed:
		52
		53	> Select SENSORID(INT), SENSORCODE(INT), SENSORTYPE(STR), METHODDESCRIPTION(STR), STATUS(CHR), LOGGERNAME(STR), MODULE(STR), CHANNEL(STR), ADDRESS(STR) from VIEW_SENSORS
		54
		55	> Select CALIBRATIONPARAMETERID(INT), PARAMETERCODE(STR), PARAMETERVALUE(DBL) from CALIBRATIONPARAMETERS where sensorid = XXXX
		56	>>This query is executed for each sensor
		57
		58	> Select SENSORTYPE(STR), VARIABLEINDEX(INT), INVARIABLEID(INT), VARIABLENAME(STR), VARIABLEUNIT(STR), NODATAVALUE(DBL), VALIDRANGEFROM(DBL), VALIDRANGETO(DBL) from VIEW_INPUT_VARIABLES_METADATA
		59
		60	> Insert into DATAVALUES (datavalue, localdatetime, sensorid, variableid) values (AAAA, to_date(‘BBBB/BB/BB BB:BB ', 'YYYY/MM/DD HH24:MI'), CC, DD)
		61	>> ~~ Where A, B, C, D are the actual values required ~~
		62
		63
		64
		65	= Measurement file format =
		66
		67	(!)This application will only process properly formatted JSON files.
		68
		69	{{include document="MeasurementFile"/}}