Last Updated on Wednesday, May 24, 2000 at 17:49:43. by Author: Nigel George

Data Tracking Proposal Pass 2

The LOG_ID effectively becomes the JOB_ID, as it uniquely identifies the JOB run

Table PHOFFLINE.LOG_ENTRIES

This table gives each job a unique key = LOG_ID.

Table PHOFFLINE.LOG_ACTIONS

would add DATA_VALIDATION Action, and any others as time goes on (eg: Production Pass1 etc..)

Table PHOFFLINE.RUNSEQ_FILES

This table associated run and sequence numbers to those files for which this makes sence. The FILE_ID is a unique reference of a file.

Table PHOFFLINE.INPUTS

This table defines which of the files are inputs, the rest are assumed to be outputs in the table of files.

Table PHOFFLINE.FILES

Make assumption that FileNames do not change in any movement of a file.The file id uniquely defines the file.

(Creation date = unix date file created, MD5_SUM= checksum i.e. for when copy etc..)

(*see table PHOFFLINE.RUNSEQ)

Note: One problem with scheme, if run over very many files (i.e. making a phi dst from many runs/seq), then this table will become huge, i.e. 1000's entries for each job, so it wont work this way. Can just make policy decision that these types of runs, will have to be handled differently. (Also could somehow link a FILE id with a FILE_TYPE=multiple, to the list of FILES, and then only get one entry for the list of inputs removing problem..)

Table PHOFFLINE.FILE_LOCATION

There is an additional table that specifies file location (also, if deleted etc..)

The status variable is 0 if current file. If file is deleted it is incremented by one, for each time the file was deleted. This way can one can tell if the file was deleted or not, one also has a history of files used.

What can happen to files, and how it fits into scheme.

a) Move a file: Simply change SYSTEM and Directory

b) Delete a file: Note, you want a record of the file you did run on, even though deleted as could have been used further down the chain..
Before remove this file

After remove this file

c) Copy a File:

To copy a file, you will require to do it via a special script that assigns to that copy a LOG_ID, and checks the MD5_SUM matches. (This is ok, because these are special files, ones that are tracked, hence they have special rules apply to them, else choas..)

Before copy this file

After copy this file

Now if want to delete original file, use again a special script that now changes status flag

Table PHOFFLINE.MODULES

This table effectively would be filled etc by Patrick with the $ Parameter scheme.

(do we need locations of modules relative to PHATHOME?)

Table PHOFFLINE.PHATJOBS

This table keeps a record of exactly what Phat and macro within that phat version were used. Note: I have considered as a matter of policy, that only Phat jobs with Phats and macros that have come from CVS are to record their information in the database. Otherwise it is impossible to track.

Table PHOFFLINE.DATABASE_KEYS

This table is so that you know exactly what files/info were used from the database. For the ped/gains etc, i.e. any job that falls within this structure (assuming pedestal jobs will be logged in a similar way etc), then you can specifically reference the log_id of that job. However the implementation of even this George has stated is quite difficult and requires major thought/work. The issue of things read in that were not created with a log_id, becomes more difficult...

Table PHOFFLINE.JOBS

This table has machine/job details

Table PHOFFLINE.DV_JOBS

Table specfically for the infomation from Data Validation Jobs

Could have additional columns as more DV specfic info needed.

Table PHOFFLINE.JOB_QUALITY

This table contains the quality information.

How to handle refinements in quality??

Odd tables that are useful

Table PHOFFLINE.RUNSEQ

This table is purely for the raw data, it is in essence filled by the DAQ for each run/seq file it makes.

Table PHOFFLINE.RUN_NUMBERS

This table's purpose is if know the run number, you can find all the log_ids relevant to a given run number.