VT ARC Facilities

Machine Room

Virginia Tech Data Center is located within the 51,000 square-foot Andrews Information Systems Building (AISB) in the Virginia Tech Corporate Research Center. This building contains office areas, the computer Data Center, a 12,000 square-foot facility that already houses the System X supercomputer as well as the university's main computing systems, and a telecommunications switch center. This building is a secure facility with emergency power, an electronic access control system, surveillance cameras, and, after normal business hours, security guards. The Data Center is protected against fire by a Halon gaseous automatic fire suppression system.

Electrical Systems

There are two separate electrical and heating, ventilation and air conditioning (HVAC) systems for the Andrews Information Systems Building. The first was installed when the building was built and provides power and air conditioning for office areas and main areas of the Data Center. The second system was installed during the construction of the System X supercomputer.

The Data Center has a separate 0.9 Megawatt electrical service entrance installed. This service entrance also has dual high-voltage power feeds and an automatic failover high-voltage switch. The power system has:

• A 0.6 Megawatt backup emergency generator and a 0.3 Megawatt uninterruptible power Ssstem (UPS) with thirty minutes of backup battery time for critical computing loads.
  
  
• Five power distribution units (PDU) used to distribute power to the equipment racks.

The Data Center has traditional downdraft standalone HVAC units. There is enough system capacity to provide redundancy if one of the units should fail. All of the HVAC equipment is supplied with redundant power by the 0.6 Megawatt emergency generator.

During the construction of the System X project, a 1.5 Megawatt electrical service entrance was installed. The redundancy of this service entrance was enhanced with dual high-voltage power feeds and an automatic failover high-voltage switch. This system is presently operating with a peak load of about 46% of its usable capacity.

The redundancy of this electrical system was increased by the installation of a 2.0 Megawatt backup emergency generator. A 1.0 Megawatt uninterruptible power system (UPS) with thirty minutes of backup battery time for System X was also installed. The UPS is currently operating at about 36% of its rated capacity.

Also included in this power system were four 225 KVA power distribution units (PDU) used to distribute power to the equipment racks. This distributed system ensures that a failure in one area will not bring the entire system down. The PDU's are currently operating at about 32% of the total rated capacity.

Cooling Systems

Utilizing the existing space in the Data Center to install System X created some interesting cooling challenges. Normal under floor cooling with downdraft HVAC units would not provide enough cooling due to the density of the computing equipment and the height of the existing raised floor (18"). Therefore, Virginia Tech worked closely with the Liebert Corporation to develop a new heat rejection system. This system required a new equipment rack layout with hot and cold aisles and new XDV HVAC units mounted to the top of each rack. There are a total of seventy XDV units in the data center. This system was supplemented with traditional downdraft HVAC units. This layout creates a great deal of redundancy since, if one of the XDV units should fail, only the equipment in that one rack would be affected. At the time of the installation, this system utilized the very latest in data center heat rejection technology.

Also as part of the System X project, redundant 125-ton chillers were installed with the provision to easily install a third chiller when it becomes necessary. This system is presently operating at about 33% of its rated capacity. All of this HVAC equipment has redundant power provided by the 2 Megawatt emergency generator.

Virginia Tech, therefore, is a unique installation with its quadruple-redundancy power system and multiple-redundant HVAC systems. These systems are designed to operate at 99.99% up-time.

External Network Connectivity

Virginia Tech is a leader in development of a regional and national optical research network infrastructure. It has successfully leveraged these efforts to ensure superior connectivity to national and international research networks using Dense Wave Division Multiplexing (DWDM) technology.

A network comprised of DWDM technology combines multiple optical carrier signals operating at different wavelengths onto a single optical fiber. This allows for a significant multiplication in capacity of the fiber resource. National LambdaRail, for example, is designed to support up to eighty 10 Gbps channels over a single pair of fiber. The VORTEX and MBC networks described below are DWDM systems and include these capabilities.

Present external network connectivity:

• Virginia Tech's VORTEX (Virginia Optical Research Technology Exchange) project provides 10 gigabit/second (Gbps) optical channels now. An existing 10 Gbps Ethernet link from the main campus in Blacksburg to the NLR DC site is operational and dedicated to research applications. Three additional 10 Gbps channels are allocated and can be activated as needed. VORTEX currently reaches National LambdaRail, Internet's Abilene Network, the Department of Energy's ESnet, the MAX network, and a NetworkVirginia gateway. VORTEX could be used to quickly reach distributed nodes in several places throughout Virginia including Norfolk, Richmond, Charlottesville, Ashburn, Lynchburg, Roanoke, and Blacksburg.
  
  
• NetworkVirginia -- An existing OC12c 622 Mbps link to the Blacksburg campus and gigabit Ethernet links to Virginia Tech's Washington area facilities supports "production" Internet, Internet2 Abilene, distance learning, Voice over IP (VoIP), and other routine services.

Virginia Tech has a robust campus network infrastructure with sufficient fiber and other resources. The high performance computing center is co-located with a major network service node, providing ease of access to national and regional research networks.

Further examples indicative of Virginia Tech's leadership roles in the networking field:

• NLR - Virginia Tech is a founding National LambdaRail (NLR) Class A Member with responsibility for facilitating all access to the NLR Washington DC node for Virginia, Maryland, and Washington DC. Virginia Tech holds a seat on the NLR Board of Directors and currently a seat on NLR's Executive Committee.
  
  
• NWVng gigaPOP - The Virginia Tech Operations Center (VTOC) operates the NWVng gigaPOP and NetworkVirginia providing aggregated access to Internet2, MAX, and other research networks for all institutions in Virginia. The VTOC is under contract to Sprint to manage the statewide core of NetworkVirginia supporting Internet access for an estimated 1.4 million people through over 600 institutions statewide.
  
  
• Mid-Atlantic Terascale Partnership (MATP) - MATP facilitates collaboration for high performance computing and networking among institutions in the region. MATP has licensed NLR network access rights from Virginia Tech on behalf of its members. The university serves as the managing member handling all MATP administration and operations. The VTOC owns and operates the DC NLR Network Aggregation Facility which provides access to NLR's Layer 2 and Layer 3 networks in McLean, Virginia.
  
  
• Mid Atlantic Crossroads (MAX) - Virginia Tech co-founded MAX and currently holds a one-quarter interest together with the University of Maryland, Georgetown University, and the George Washington University. MAX provides Washington, DC metropolitan area optical transport services and interconnection among higher education and federal research networks and facilities.
  
  
• Mid Atlantic Broadband Cooperative (MBC) - Virginia Tech provided the impetus for creation of this non-profit cooperative which is currently constructing over 700 miles of fiber optic infrastructure throughout Southside Virginia. The university is working to procure Indefeasible Rights to Use (IRU) dark fiber through NLR and Internet2 FiberCo on multiple paths from Washington to Atlanta intersecting MBC's fiber at multiple points. Virginia Tech and MBC will jointly build a high performance Dense Mode Wave Division Multiplex (DWDM) network system linking the MBC network and Virginia Tech to several points including Washington, DC, Raleigh, and Atlanta. This system will provide Virginia Tech with multiple 10+ Gbps channels to multiple city nodes. The project includes extensive metropolitan area dark fiber systems in DC/Northern Virginia and Atlanta to extend this connectivity to multiple points of interest in those areas.