We need more bandwidth. After technical discussions with our data center hosting+ISP, green.ch, we opted to run a 10 GbE fiber optic cable into our rack, rather than expand our 2Gbps LACP bond to 4Gbps which would have only been an intermediate step towards 10 GbE. In the long run, this was the more cost effective, lower effort decision.

Having not worked with fiber before, it's been an educational process. A couple things I've learned that mildly surprised me:

• There are a couple of standards for 10 GbE fiber optic networking, mainly differentiating by how long the cable run is.
• Fiber optic transceivers are crazy expensive--easily equally or more pricy than CPUs by both size and weight. I'm glad we don't need a long run!
• Our switch allows hot plugging of modules and transceivers. If I had known this before it would have saved me a late night trip to the data center.

Doing so required upgrading our switch, an HP 2910al-g24 (J9145A) with an expansion module and a transceiver. The expansion module, an HP 2-port 10GbE SFP+ al Module (J9008A) inserts into the back of the 2910al. And the fiber optic transceiver, an HP X132 10G SFP+ LC SR Transceiver (J9150A) inserts into the back of the module.

The switch has two ports for modules, and each module has two transceiver ports. We still have room to expand! A 20 GbE LACP'd Internet connection in the coming years is not out of the question.

Great stuff.

Add a comment

Posted by Stu Thompson on 11 April 2011 13:47:00 CEST #

I built a new streaming server recently. It is amazing how much computing power one can buy these days with a modest amount of cash. We are expecting throughput in the 4Gbps region with this machine, and initial benchmarks support this figure.

Hardware

• Intel Core i7 990X 6-core, 12-thread 3.4GHz (burst to 3.73GHz)
• 24GB (6x4GB) DDR3-1066 memory
• ASRock x58 Extreme 3 mobo
• (2) Intel PRO/1000 PT Dual Gigabit network adapters
• (2) 160GB 2.5" 5400-7200 RPM drives in a RAID-1 software mirror
• SuperMicro 2U SuperChassis 523L-410B
• Cold Cathode Fluorescent Lamps, green
• Low end, passively cooled, half-height PCI video card

Software

• Debian 6 Squeeze
• Sun Java 6
• Wowza 2.3

One pesky little problem that came up was the size of the heat sink. It did not fit into the 2U rack mount chassis, as did a similar previous build with a lower end LGA1156 4-core i7. It kind looks like a hotrod from the sixties! After some shopping around I sourced a lower profile fan that fit into the case. With such a "hot" processor (135 watts) there is a bit of concern about heating. Some CPU load testing with openSSL (all 12 threads running ~100% for an hour) with no problems indicates that all should be good. Time will tell.

Last week I installed zimbali into the rack at our data center. These past few days I've been finalizing the configuration and benchmarking it. We hope to have it in production next week.

Yes, those are CCF (Cold Cathode Fluorescent) lamps in the chassis. Beside just looking cool, they do serve a purpose. Since zimbali is built with mostly consumer-grade hardware, outside of the chassis and NICs, there is poor monitoring of the physical system. E.g.: temperature, fans, etc. With the CCF lamps installed I can eyeball the state of the fans on my visits to the data center. Unconventional, but oh-so efficient and economical.

Responses[2]

Posted by Stu Thompson on 31 March 2011 10:00:00 CEST #

While slumming around meta.stackoverflow.com a year back, I picked up a fetish for waffles, hand-drawn diagram annotations, and ponies. Unicorns are cool too. This odd fetish has manifested itself in the server room lately. All of the kit is adorned with little pony stickers I picked up at the local Jumbo hardware store a while back.

Click teh image for a huge version. (1MB+)

           

Add a comment

Posted by Stu Thompson on 22 March 2011 17:20:00 CET #

I recently moved my Pebble 2.3.1 blog from one machine to another. In the process of doing so a couple of things broke, like comment validation. Trying to review and approve comments resulted in a mostly blank screen.

A check of the logs gave me my first clue:

Mar 22, 2011 2:24:37 PM org.apache.catalina.core.ApplicationDispatcher invoke
SEVERE: Servlet.service() for servlet jsp threw exception
org.apache.jasper.JasperException: /WEB-INF/jsp/viewResponses.jsp(65,0) "${response.class.name == 'pebble.blog.Comment'}" 
contains invalid expression(s): javax.el.ELException: [class] is not a valid Java identifier
	at org.apache.jasper.compiler.DefaultErrorHandler.jspError(DefaultErrorHandler.java:40)
	at org.apache.jasper.compiler.ErrorDispatcher.dispatch(ErrorDispatcher.java:407)
	at org.apache.jasper.compiler.ErrorDispatcher.jspError(ErrorDispatcher.java:198)
 {snip}

My Google searches quickly lead me believe other people, not all of them Pebble users, were having a similar issue. And folks were blaming the more recent releases of Apache Tomcat. As it turns out, the Tomcat developers changed the default value for org.apache.el.parser.SKIP_IDENTIFIER_CHECK from true to false. What it does is tell the JVM that identifiers should not be checked to ensure that they conform to the Java Language Specification for Java identifiers. This was done to more closely comply with the JLS. Unfortunately, it has lead to many JSP-based webapps breaking.

With Pebble, the problem was the use of JLS reserved keywords for identifies. In the below snipped from WEB-INF/jsp/viewResponses.jsp, Pebble uses the reserved word "class".

  <c:if test="${response.class.name == 'pebble.blog.Comment'}">

If you are a developer of a webapp that is experiencing this issue, it is best if you just work through the problems and change your app. If you are a plain old users like me, try upgrading...or adding this system property to your Tomcat startup script. (E.g.: {tomcat-home}/bin/startup.sh) and restart.

JAVA_OPTS="$JAVA_OPTS -Dorg.apache.el.parser.SKIP_IDENTIFIER_CHECK=true" 
export JAVA_OPTS

And presto! Problem solved. Hopefully. For now. Until it breaks again. Because this is a workaround, not a solution.

Responses[4]

Posted by Stu Thompson on 22 March 2011 15:28:00 CET #

In the name of "Keep it simple, stupid!" I don't install the X Windowing system onto my Debian Lenny and Squeeze servers. (And hence no Gnome, no KDE.) What I do change is the screen resolution from the default 640x480 to 1024x768, which any modern monitor can support. This has several advantages:

• The boot sequence and commands like ls -l will show 56% more lines: 30 vs. 47
• Commands like ps aux won't wrap as frequently
• The huge fonts in text-mode 640x480 are just plain ugly
• Increases the overall number of characters displayed on a single screen by ~2.5x

Step one: Set the resolution for Grub 2. Edit /etc/defaults/grub with your favorite editor. Uncomment the GRUB_GFXMODE line and change the value to 1024x768.

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
GRUB_GFXMODE=1024x768

Step two: Have the resolution passed to the kernel when it is loaded. Find the below block around line 126 in /etc/grub.d/00_header and add the set gfxpayload=keep line. I've also commented the edit, but that is optional.

if loadfont `make_system_path_relative_to_its_root "${GRUB_FONT_PATH}"` ; then
  set gfxmode=${GRUB_GFXMODE}
  #keep the gfx mode set in grub (stu)
  set gfxpayload=keep
  load_video
  insmod gfxterm
fi

Step three: Generate a grub2 config file based on the template edits above.

root@zimbali:~# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-5-amd64
Found initrd image: /boot/initrd.img-2.6.32-5-amd64
done

And that's it! On the next boot the resolution will be greater and the text terminal will display many more characters.

Responses[5]

Posted by Stu Thompson on 01 March 2011 09:05:00 CET #