Process NIH GEO GSE data by geoQuery

How to get an Expression value table of a GSE* file from GEO website.

for example: GSE33147

>g = getGEO("GSE33147")

......

it may download a series data matrix file: GSE33147_series_matrix.txt.gz
then load the dat again:
>g = getGEO(filename="GSE33147_series_matrix.txt.gz")

check the data

>class(g)

get the ExpressionSet:

> e = as(g, "ExpressionSet")

get the data table
> f = exprs(e)

save:
> write.csv(f, file="***")

load the group gene names that you want: (assume you only want part of them)
the names are stored in file "top60.csv"
>genes = read.csv("top60.csv",header=T)

the genes are factors, we need change them to character,
> cgenes = as.character(genes[,1])               //the first column.

>

Processing Microarray data by using R and Bioinductor

Load all the .CEL.gz file from a folder:

>library(affy)
>data <- ReadAffy()

## For Affymetrix data, there is no concept of RAW data.
See page 72 of DNA microarray data analysis using Bioconductor.

Now get the RMA data
> data.rma <- rma(data)

or you can use the following if the size of data is too large.

>data.rma <- justRMA(data)


change the RMA result to expression values
data.e <- exprs(data.rma)

save the data.e
write.csv(data.e,file="data.csv")

##now process the data and map probe ID to gene ID
##by any script language ....such as python


now load the processed data to R again:
d <- read.table("processed_data.csv",header=T,sep=",")

now d is a string matrix, make it to float

df <- data.frame(d,row.names=1)    ## use the first column as name

calculate the mean and append to df

df$mean <- rowMeans(df)

ranking

dfr <- df[order(-df$mean),]

save the highest 60 to a .csv file

write.csv(dfs[1:60,],file="d60.csv",sep=",")

upgrade Ubuntu OS

sudo apt-get install update-manager-core
sudo vi /etc/update-manager/release-upgrades
change the line below from prompt=lts to prompt=normal.
sudo apt-get update && sudo apt-get upgrade
sudo do-release-upgrade

Multiple interface on same network are using same NIC for communication

Multiple interfaces on the same subnet

In the Linux implementation of the IP stack a IP address belongs to the host event though the administrator configures it on a devices. This can cause somewhat unexpected behaviour when multiple interfaces are configured to use the same network.

The network

 {network A}
            \       +--------------+
             -(eth0)| Linux server |
                    +--------------+
                     (eth2)  (eth3)
                       |       | 
                      {Network B}

When multiple interfaces are configured for the same network you must use policy routing to make the internal IP stack route the packages out on the designated interface. This is done by using the "ip route" command.

Prerequisites

The following options must be enabled in the kernel.

CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_NETLINK_DEV=y

You also need the iproute suite, also known as iproute2. In Debian (and Debian derivatives) this is the iproute package.

Example configuration for two interfaces on the same IP subnet

In Debian (and Debian derivatives) the easiest way to add the additional routes on start-up is to use the up option in /etc/network/interfaces.

auto eth2
iface eth2 inet static
   address 192.168.1.20
   network 1192.168.1.0
   netmask 255.255.255.0
   broadcast 192.168.1.255
   up ip route add 192.168.1.0/24 dev eth2 proto kernel scope link src 192.168.61.20 table 20
   up ip route add default via 192.168.1.1 dev eth2 table 20
   up ip rule add from 192.168.1.20 lookup 20
 
auto eth3
iface eth3 inet static
   address 192.168.1.21
   network 192.168.1.0
   netmask 255.255.255.0
   broadcast 192.168.1.255
   up ip route add 192.168.1.0/24 dev eth3 proto kernel scope link src 192.168.61.21 table 30
   up ip route add default via 192.168.1.1 dev eth3 table 30
   up ip rule add from 192.168.1.21 lookup 30

Note:The table id is just a positive integer in the range 0-255 that identifies a unique table. When setting up multiple interfaces on the same subnet this id needs to be unique for each interface. In the example the interface number times 10 is used. Table id 0 and 253-255 are reserved for internal use and may not be used for this configuration.

For more information about advanced Linux routing please see read the Linux Advanced Routing & Traffic Control HOWTO.

Book tip: "Linux Network Internals"

another example:

Multiple interface on same subnet same machine can work fine. 
we need to use advance routing concepts with arp_filter option, make table for each interface and configure default routes and lookups. following three magic lines solved my problem.

Code:

ip route add 10.209.192.0/22 dev eth1 proto kernel scope link src 10.209.193.131 table tlb_1
ip route add default via 10.209.192.1 dev eth1 table tlb_1
ip rule add from 10.209.193.131 lookup tlb_1
ip route flush cache

install torque client

 cd /software/
  934  ls
  935  cd torque/
  936  ls
  937  cd 4.1.2/
  938  ls
  939  ./torque-package-mom-linux-x86_64.sh --help
  940  ./torque-package-mom-linux-x86_64.sh --listfiles
  941  ls
  942  ./torque-package-clients-linux-x86_64.sh --listfiles
  943  ./torque-package-clients-linux-x86_64.sh --install
  944  clear
  945  qsub
  946  exit
  947  df -h
  948  cd /software/
  949  ls
  950  cd torque/
  951  la
  952  ls
  953  cd 4.1.2/
  954  ls
  955  ls -ltr
  956  pwd
  957  cd /software/toolss
  958  cd /software/tools
  959  ls
  960  cd ..
  961  ls
  962  cd torque/
  963  ls
  964  cd 4.1.2/
  965  ls
  966  ls -ltr
  967  sh ./torque-package-clients-linux-x86_64.sh
  968  sh ./torque-package-clients-linux-x86_64.sh --install --verbose
  969  ls
  970  cd ..
  971  ls
  972  cd ..
  973  ls
  974  pwd
  975  cd src/
  976  ls
  977  cd torque-4.1.2
  978  ls
  979  cd con
  980  ls
  981  cd contrib/
  982  ls
  983  cd init.d
  984  ls
  985  ls -l /etc/init.d|grep trq
  986  cp -a trqauthd /etc/init.d/
  987  cd /etc/init.d
  988  ls -ltr
  989  getent passwd
  990  ls
  991  service trqauthd start
  992  qstat
  993  pbsnodes -l
  994  chkconfig --add trqauthd
  995  chkconfig trqauthd
  996  chkconfig --list|grep trqauthd
  999  vim /etc/hosts



//modify the torquemaster, and add new host to the manager by "qmgr -c"

specified unpartitioned disk sda in partition command

When getting this error message in a kickstart;

specified unpartitioned disk sda in partition command

Its possibly due to dmraid information still present on the disk from an earlier installation. Verify by running (press F2 after kickstart halts):

$ dmraid -r -D /dev/sda

You should get information about the disk being member of a raidset.

The inforamation can be stored in different locations on the harddrive, to my knowledge its commonly in the end though. If you dont want to consider this further, you can delete the entire disk by running;

$ dd if=/dev/zero of=/dev/sda

It will take some time as the entire disk is being written to. If you want to make a more surgical approach, just erase the last couple of sectors. First run fdisk;

$ fdisk -l
Disk /dev/sda: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 byte

Then run the following command. Adapt the values for ”bs” and ”seek” to suit your actual harddrive. You get the values from the fdisk output above. In the below example, i’m deleting the last 10 sectors of the disk (91201 – 10 = 91191):

dd if=/dev/zero of=/dev/sda bs=8225280 seek=91191 count=1

Verify that the dmraid information is actually gone;

dmraid -r -D /dev/sda

The output should now be different from before, saying the disk is not a member. If so, you should be able to use it to kickstart on. 

specified unpartitioned disk sda in partition command

When getting this error message in a kickstart;

specified unpartitioned disk sda in partition command

Its possibly due to dmraid information still present on the disk from an earlier installation. Verify by running (press F2 after kickstart halts):

$ dmraid -r -D /dev/sda

You should get information about the disk being member of a raidset.

The inforamation can be stored in different locations on the harddrive, to my knowledge its commonly in the end though. If you dont want to consider this further, you can delete the entire disk by running;

$ dd if=/dev/zero of=/dev/sda

It will take some time as the entire disk is being written to. If you want to make a more surgical approach, just erase the last couple of sectors. First run fdisk;

$ fdisk -l
Disk /dev/sda: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 byte

Then run the following command. Adapt the values for ”bs” and ”seek” to suit your actual harddrive. You get the values from the fdisk output above. In the below example, i’m deleting the last 10 sectors of the disk (91201 – 10 = 91191):

dd if=/dev/zero of=/dev/sda bs=8225280 seek=91191 count=1

Verify that the dmraid information is actually gone;

dmraid -r -D /dev/sda

The output should now be different from before, saying the disk is not a member. If so, you should be able to use it to kickstart on.