This morning while attempting to tether my brand new iPhone 4 32GB to the almost new 2010 Unibody MacBook Pro, I ran into a series of problems. (Why do this? Because I want to create a PAN, or a Personal Area Network. This will allow me to use my iPhone 4 for wireless Internet access while out and about.) After four attempts at pairing the two in an effort to enable tethering, I had to hit the intertubes to find a fix. Unfortunately, others had similar problems.

So, Navigating to Settings >> General >> Network >> Internet Tethering and flipped the switch to 'on'.

Then it was time to initiate the pairing from the MacBook Pro. Now to start the pairing on the iPhone. Navigate to Settings >> General >> Bluetooth and the phone automagically starts searching for the MBP. Poof! Once the Mac sees the iPhone it gives you a security number to ensure it is syncing to the same devices, like here:

You see the same security number on the iPhone, and hit the Pair button.

Much to my chagrine, the below "Pairing Unsuccessful" message showed up each time I tried to pair the devices.

After the fourth attempt, I checked Console to see if there were any interesting messages.

  Bluetooth Setup Assistant[425]  starting pairing
  Bluetooth Setup Assistant[425]  Pairing failed with 'Stu Thompson’s iPhone': error 0x10

Yup, that's interesting. Out in the Googleverse there are plenty of folks having all sorts of problems pairing their iPhones and MacBook Pros, in an attempt to tether or otherwise. There is information about updating the Bluetooth firmware, but that was not relevant to my brand-spanking-new MBP. So, after some peeking and poking, I decided to delete the existing Bluetool PAN on the MBP.

And that worked! (To delete the old PAN, just select "Bluetooth PAN" in the Network Preferences and then click the little - button at the bottom left.)

What I had failed to take into account was that there was a previous tethering relationship configuration in the original Bluetooth PAN. Several weeks ago my buddy Lincoln and I had paired my MBP to his iPhone so that we could get on the net while discussing some technology mumbo jumbo at Imagine.

A little bit frustrating. Although this is an edge case situation, Apple could and should make the tethering process more robust. Responsible Stu thinks he should file a report with Apple. Pessimistic Stu thinks communicating with Apple is too opaque and unrewarding. Stu's wife thinks he should stop geeking this Sunday morning and pay more attention to her. Two against three: no bug report.

Add a comment

Posted by Stu Thompson on 28 November 2010 10:31:00 CET #

import java.util.concurrent.atomic.AtomicInteger;

public class ThreadCounter extends Thread {
	private static final AtomicInteger count = new AtomicInteger();
	
	public static void main(String[] args) {
		while (true)
			(new ThreadCounter()).start();
			
	}

	@Override
	public void run() {
		System.out.println(count.incrementAndGet());
		
		while (true)
			try {
				Thread.sleep(1000);
			} catch (InterruptedException e) {
				break;
			}
	}
}

  javac ThreadCounter.java
  java  -cp . ThreadCounter

The results:

Interesting...

Responses[2]

Posted by Stu Thompson on 24 November 2010 21:27:00 CET #

Six months after starting this project, I received an email from Ryan Ellis asking me if I had any tips regarding this NAS build.

Just read your article on the NAS. Amazing job. I would like to replicate what you did. Any tips?

Well, I most definitely do have a list of things I'd have done different! Nothing too dramatic, but there are some decisions that were overkill and some where I went a little too low end.

• Using Intel and Asus motherboards: The Intel DP55KG w/ P55 Express chip set is not liking the Ubuntu 10.04 LTS Linux, or apparently any Linux distro for that matter. Specifically, the NAS box with the Intel mother board is unable to do a soft reboot. That means every reboot requires my physical presence in the data center. This has been a known problem for a while but it did not turn up during my mobo research. Many folks have tried various kernel options to change the rebooting behavior with mixed success. I've not been able to resolve the issue. When building up the NAS box I told myself that the Linux community would eventually resolve the issue. Maybe it has, but now that we are in production I can't really experiment with the server.
  
  Lesson Learned: If the mobo is not working perfectly for you, then find another. It's too painful to revisit once in production.
  
  
• Not using "server grade" motherboards: Linux is unable to monitor things on the Asus and Intel motherboards, like fan speed and temperature, that I'd like to be graphing in Cacti. This is apparently possible with the "server grade" budget motherboards from the likes of SuperMicro.
  
  
  Lesson Learned: It only saved us SFr.400-800 to use these performance desktop motherboards, but our ability to proactively monitor fans is lost. I wish I'd gone for a SuperMicro motherboard.
  
  
• The network load is much lower than I had realized, so the Intel Quad-port NIC is overkill--not even 100 Mbps at peak usage! This is apparently due to the client side file cache on our client server machines. This was difficult to predict on our old system because we were running with direct attached storage. In hind sight I wish I'd done more research. The two Intel PRO/1000 PT Quad Port Server Adapter could have been single port NICs, saving us SFr.800 total.
  
  
  Lesson Learned: Try to accurately measure and predict how much network traffic you'll see. Did I really need four port NIC bonding? No even close.
  
  
• I didn't pay enough attention to adapter-to-drive cabling. The LSI 3ware 9650SE-ML16 card came with 1-to-4, Multilane-to-SATA breakout cables, but the SuperMicro SuperChassis 836A-R1200B came with backplane with four Multilane ports. That ment sourcing four CBL-SFF8087-05M Multilane-to-Multilane cables, an extra cost. And when I did get them, two were ~10cm shorter than I would have preferred--the cables are currently a bit tight and cannot be moved without loosening the connection. We probably spent another SFr100 on extra cabling.
  
  Lesson Learned: At least think about device-to-device cabling beforehand, and don't leave until the build.
  
  
• RAID 1+0 may have been overkill, RAID 6 performance would probably have sufficed. Our production metrics seem to indicate that we run at no more than 33-40%, conservatively, of capacity at peak, and the vast build of our NAS activity is reads. RAID 6 probably would have been a safe choice, and doing so would have reduced the number of hard drives by 6 total (3 on each server), which would also have allowed us to use a smaller chassis. Total savings would have been SFr 1700-2000, a non-trivial amount.
  
  My wife, Robyn, helping me build up one of the NAS servers
  That said, we would be reducing our margin for error, room for future growth (there are currently two empty drive bays on each server), and not allowed changes in application behavior which would result in more writes. (RAID 6 is great for heavy read applications, like ours, but have much weaker write performance characteristics.)
  
  
• I did not appreciate how little I understood drbd, or block-level replication for that matter. This resulted in taking poorly understood actions on production data. In hind sight, it would have been wise to setup a test environment on the side (Amazon EC2, some old kit, whatever) and experimented. If I had made a mistake, we would have had to implement our disaster recovery procedures, which are time consuming and resulted in non-trivial down time.
  
  Lesson Learned: If it works like magic, then you don't have a clue how it works. For something as fundamental as DRBD is to a redundant NAS system, one should make decisions ad novum, 'with intent'.
  
  
• Setting up the monitoring was significantly more work that I had predicted. While our Cacti + SNMP setup is very powerful, it is not easy to get going for anything but very common metrics. Specifically, configuring important alerts for things like drive failures, or graphs of NFSv4 metrics has been a considerable amount of work. In fact, I've had to come up with my own NFSv4 Cacti template which, to my surprise, did not exist.
  
  
  
• These boxes are heavy. Like in the 30kg region. Installing them into the rack alone, even with the assistance of a foot-actuated hydraulic lift, was difficult and borderline dangerous. Managing to get the rails aligned correctly was very challenging.
  
  60kg of Network Attached Storage
  
  Lesson Learned: Don't install anything other than a switch alone.
  
  
• WD Green versus WD RE4 drives: We could probably have used cheaper WD Green drives instead of the RE4 series "Enterprise Hard Drives". Ryan Shrout and Patrick Norton talk about the apparent fallacy that WD Green drives are not suitable for a NAS in Episode #95 of This Week in Computer Hardware. The cost savings is huge. Currently at Digitec.ch, where we bought our drives, a WD Caviar RE4 2TB runs for SFr255 and a WD Caviar Green is SFr109--a SFr146 savings. With the 22 data mount hard drives in our build, that works out to SFr3,212! And we could have saved an additional SFr~168 on the operating system drives too.
  
  

All of that said, we are in production and everything works. More dramatically, this project after a mere six months has already resulted in a positive return on investment, when accounting for hardware costs alone. Factor in the time I spent on this project, 60-80 hours, and we will be in the black some time in Q1 2011. Not bad. (This self-built approach was taken in favor of outsourcing our storage to our hosting company's shared NetApp NAS at a TB/month rate.) It also has been a wildly educational experience and forced me to understand my application even more than before.

Add a comment

Posted by Stu Thompson on 21 November 2010 21:13:00 CET #

This morning we discovered that the MPEG-TS streams our application was producing did not meet Apple's latest, seemingly every changing, requirements for playback on the iPhone and iPad. Never mind that our streams playback just fine; Apple now requires that they get a stamp of approval from Media Stream Validator Tool (mediastreamvalidator) for new iPhone apps. To get mediastreamvalidator, I downloaded and installed it. Apparently it is still in beta.

mediastreamvalidator, when run with arguments let's us know what the commandline line arguments and options are.

manoa:~ stu$ mediastreamvalidator
Media Stream Validator
http://www.apple.com

Basic commands:
  mediastreamvalidator help       list all commands or get more help on a command
  mediastreamvalidator parse      parse a playlist
  mediastreamvalidator selftest   run the mediastreamvalidator test suite
  mediastreamvalidator validate   validate a playlist

manoa:~ stu$ mediastreamvalidator --version
Media Stream Validator: Beta Version 1.0(101102)
  Python standard library: /System/Library/Frameworks/Python.framework/Versions/
        2.6/lib/python2.6
  msvlib: /Library/Python/2.6/site-packages/msvlib

Copyright 2009-2010 Apple Inc.
http://www.apple.com

manoa:~ stu$ 

Interesting, it has a selftest. Let's try that out!

manoa:~ stu$ mediastreamvalidator selftest
test_parse_path (test_utilities.URLUtilitiesTests) ... ok
                        {snip}
test_correct_sliding_sequence 
  (test_server_validator.ServerSlidingWindowCorrectBehaviorTests) ... FAIL
                        {snip}
(test_server_validator.ServerVariantPlaylistTests) ... ok

======================================================================
FAIL: test_correct_sliding_sequence (test_server_validator.ServerSlidingWindowCorrectBehaviorTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Python/2.6/site-packages/msvlib/tests/test_server_validator.py", 
    line 1054, in test_correct_sliding_sequence
  File "/Library/Python/2.6/site-packages/msvlib/tests/test_server_validator.py", 
    line 76, in check_valid
  File "/Library/Python/2.6/site-packages/msvlib/tests/test_server_validator.py", 
    line 40, in _check_valid_object
AssertionError: Unexpected fatal parsing problems:

ERROR: First line of playlist must be an M3U tag.
1:    
      ^


----------------------------------------------------------------------
Ran 189 tests in 199.610s

FAILED (failures=1)
manoa:~ stu$ 

A failure. Great. Now what do we do? Is the validator valid? Is the validator's analysis of our media stream valid?

Producing media for the iPhone is getting harder, more complicated. One would expect it to become easier, faster, better, safer. Our streams are being held hostage by beta software that fails a self test from Apple.

(OK, self pity drama queen rant over. Back to work.)

Add a comment

Posted by Stu Thompson on 17 November 2010 09:09:00 CET #

A few weeks ago, my trusty 2.5 year old 2008 MacBook Pro died. Since the machine is at the center of my working life at xtendx, it needed immediate replacement. We quickly ordered a new MacBook Pro. It sports a 2.4 GHz Core i5 CPU and 4GB of ram for only SFr 2200 from Digitec.

The only weakness was the stock 5,400 RPM HDD. Fortunately for me, the ten-month old Intel X-25M SSD inside the 2008 MBP was still good. I swapped it out inside 60 seconds! (Ha ha.)

I have to say, all that bitching and moaning on the Intertubes about the new unibody MBP is a bunch of hogwash. Opening and working inside the case of the unibody MBP is soooo much easier than on the 2008 MPB.

• Fewer screws to remove and replace (and lose)
• Less time opening and closing the case
• No need to unplug (and potentially damage) the keyboard
• No little panels to replace, like the old MBP's memory cover

I am very pleased with the result. As a side bonus, there was no need to reinstall all my software. The only complaint was from the Audible items in iTunes, for which I will have to reauthorize.

Add a comment

Posted by Stu Thompson on 04 November 2010 14:24:00 CET #