some work on entropy 330k data as of 9-15-13

2014-08-29

++ some work on entropy 330k data as of 9-15-13

I started exploring some of the entropy results.
F:\kurt\storage\CIM Research Folder\DR\2013\9-13-13\entropy

Summary spreadsheet of entropy values for the 330k data here
"F:\kurt\storage\CIM Research Folder\DR\2013\9-10-13\entropy\summary of entropy values 9-10-13.xlsx"

Here's an inkscape file with various graphs.
"F:\kurt\storage\CIM Research Folder\DR\2013\9-13-13\entropy\Graphs of 330K data 9-13-13.svg"
^make sure to find most recent version
-graphs present
--Entropy wafer 22 (Classification 1)
--Entropy wafer 22 (Type)
--CV wafer 22 (Type)
--CV wafer 22 (Classification 1)
--Normalized Entropy wafer 22 (Classification 1)
--Normalized Entropy wafer 22 (Type)
--Normalized CV wafer 22 (Type)
--Entropy wafer 46 (Type)
--Entropy wafer 46 (Classification 1)
--cv wafer 46 (Type)
--cv wafer 46 (Classification 1)
--normalized entropy wafer 46 (Type)
--Normalized Entropy wafer 46 (classification 1)
--Normalized CV wafer 46 (classification 1)
--Normalized CV wafer 46 (type)

-parameters for dot plot
--set group and type
--alpha at 0.5
--binwidth at 0.01
--dotsize at 3.0

-------------

some of the samples were incomplete because the saguaro supercomputer did not properly calculate the entropy or cv numbers for some of the samples (some folders were missing the gpr file, some results just had 0s or NaNs, etc.)

I considered going back and having Saguaro run all of the samples that didn't run well, but this could become complicated and time consuming so I think I'll just settle with the data that I have.
^Actually I'll go back and rerun the few good samples that I missed with wafer 46.  Many of them were F635 instead of F531 (or F532 I forget) samples, and two of them were just titled red green something, and I'm not sure what that is.
Here is the key for samples that didn't run the 1st time (blank means there should be no problem)
{
101	F635 Median
82
83	F635 Median
85	F635
86	
87	F635
88	
89	F635
91	F635
93	F635
94	
95	F635
98	F635
99	F635
84	green red p50 sample
90	green red P50 sample
}
list of 12 samples to rerun
{
42	4-46 S4 H3 Hi P20 111312 DTRA 1 S.gpr
43	4-46 S5 A1 Hi P20 03252013 ND84 L.gpr
86	4-46 S8 B2 Hi R-P50 G-P20 03012013 Normal 154_0532 A.gpr
88	4-46 S8 B3 Hi R-P50 G-P20 03012013 GMB OOKN_0532 A.gpr
94	4-46 S8 D3 Hi R-P50 G-P20 03012013 Mening XX_0532 A.gpr
100	4-46 S8 G2 Hi R-P50 G-P20 03012013 EC 722078_0532 K.gpr
102	HT4-46 S1 B2 Hi P20 103112 #1-1 S.gpr
104	HT4-46 S1 D1 Hi P20 103112 #4-7 K.gpr
105	HT4-46 S1 E1 Hi P20 103112 BC040 K.gpr
106	HT4-46 S1 F1 Hi P20 103112 ND145 1-50k L.gpr
108	HT4-46 S1 G2 Hi P20 103112 DM1A 500.0 pM S.gpr
82	4-46 S8 A3 Hi R-P50 G-P20 03012013 EC 712707_0532 K.gpr

}

In order to start this job, I need to modify the cmd_original.saguaro file from Josh.

The path to the files is now:
/home/lwang138/entropy/input/rerun_of_46

Example of one command:
java  -jar  "/home/lwang138/entropy/input/EntropyOfArray090413_2.jar" /home/lwang138/entropy/input/rerun_of_46/42" "4-46 S4 H3 Hi P20 111312 DTRA 1 S.gpr" -Xmx6144

What were some of the last commands that were in the terminal?
-qstat -n
-mybalance
-cd /home/lwang138/kurt
-chmod +x jobScript.saguaro
-./jobScript.saguaro cmd.saguaro pbs.saguaro

Here are the commands I will issue now to run these remaining files.
-cd /home/lwang138/kurt/2nd_run_9-14-13
-chmod +x jobScript.saguaro
-./jobSc-ript.saguaro cmd_original.saguaro pbs.saguaro

There was some problem when I tried to run the program.

Here is the list of jobs
{
6154356.newmoab            jobTemp.saguaro  lwang138        00:00:00 C serial         
6154357.newmoab            jobTemp.saguaro  lwang138        00:00:00 C serial         
6154358.newmoab            jobTemp.saguaro  lwang138        00:00:00 C serial         
6154359.newmoab            jobTemp.saguaro  lwang138        00:00:00 C serial         
6154360.newmoab            jobTemp.saguaro  lwang138        00:00:00 C serial         
6154361.newmoab            jobTemp.saguaro  lwang138        00:00:00 C serial         
6154362.newmoab            jobTemp.saguaro  lwang138        00:00:14 C serial         
6154363.newmoab            jobTemp.saguaro  lwang138        00:00:14 C serial         
6154364.newmoab            jobTemp.saguaro  lwang138        00:00:00 C serial         
6154365.newmoab            jobTemp.saguaro  lwang138        00:00:00 C serial         
6154366.newmoab            jobTemp.saguaro  lwang138        00:00:00 C serial
}

The message for the first job is this:
/var/spool/torque/mom_priv/jobs/6154356.newmoab.local.SC: line 7: unexpected EOF while looking for matching `"'
/var/spool/torque/mom_priv/jobs/6154356.newmoab.local.SC: line 8: syntax error: unexpected end of file

The command looked like this:
java  -jar  "/home/lwang138/entropy/input/EntropyOfArray090413_2.jar" /home/lwang138/entropy/input/rerun_of_46/42" "4-46 S4 H3 Hi P20 111312 DTRA 1 S.gpr" -Xmx6144
^It looks like I had an extra quotation that I did not need.

I tried running again, but I ran into another problem.  I forgot that .gpr should not be a part of the filename.  I deleted the .gpr from the commands and tried running again.

azim58wiki: