Who Owns Your Social Graph?

7 Jul

Image representing Facebook as depicted in Cru...

Image via CrunchBase

There’s a story on this issue on GigaOm, which is worth reading for context.

I don’t think Facebook owns my friends. Do you think they own yours? If you think Facebook should re-enable the “Facebook Friends Exporter” extension, or—better still—simply allow people to use their own contacts as they please, “Like” this posting. Send Facebook a message.

NEWSFLASH: World Doesn’t End For Another Ten Days!

22 May

The Four Horsemen of the Slightly-Belated Apocalypse (A. Dürer)

With all of my friends laughing it up over the failure of the much-anticipated apocalypse to appear yesterday, I was rereading some of the stuff on Camping’s (various) predictions, and I came across his statement that, “From April 1, 33 to April 1, 2011 is 1978 years”.

But it isn’t: it’s 1978 years less ten days.

In 1582, Pope Gregory XIII instituted our modern calendar, now known as the Gregorian calendar, as a replacement for the then-used Julian calendar (named for Julius Caesar). The problem with the Julian calendar was that it assumed that the year was exactly 365.25 days long, an annual error of about eleven minutes. By 1582, the spring equinox was falling as early as the 11th of March, thanks to a millennium or so of accumulated error. Several adjustments to leap years were made, but the main gross adjustment was to declare that March 11, 1582 was now March 21, 1582, by Papal decree.

Seems as though Camping forgot about this in his calculations (he may know the Bible better than I do, but I apparently know calendars better than he does). The world’s not ending until May 31st.

Oh, ye of little faith! If you were all excited about the Rapture yesterday and all depressed this morning, some angel is putting your name on a list right now, you betcha.

Motivation

28 Apr

Alfred Hitchcock was directing a film once when a member of the cast, a “Method actor”, came running up to him, waving a script.

“Mr. Hitchcock, I can’t play this scene!” the actor cried. “I don’t understand it? Why would my character say and do this? What’s my motivation?”

“Your paycheck,” replied Hitchcock.

Did Apple’s iPhone Track Lady Gaga to William and Kate’s Wedding?

27 Apr

No, it didn’t. Apple made a statement confirming what I’d been suspecting: the consolidated.db file is a cache of cell tower and WiFi hotspot locations to speed up triangulation in Location Services on iOS, period.

“The iPhone is not logging your location. Rather, it’s maintaining a database of Wi-Fi hotspots and cell towers around your current location, some of which may be located more than one hundred miles away from your iPhone, to help your iPhone rapidly and accurately calculate its location when requested. Calculating a phone’s location using just GPS satellite data can take up to several minutes. iPhone can reduce this time to just a few seconds by using Wi-Fi hotspot and cell tower data to quickly find GPS satellites, and even triangulate its location using just Wi-Fi hotspot and cell tower data when GPS is not available (such as indoors or in basements). These calculations are performed live on the iPhone using a crowd-sourced database of Wi-Fi hotspot and cell tower data that is generated by tens of millions of iPhones sending the geo-tagged locations of nearby Wi-Fi hotspots and cell towers in an anonymous and encrypted form to Apple.”

So, if you’re worried about Apple tracking you,

“Apple is not tracking the location of your iPhone. Apple has never done so and has no plans to ever do so.”

I’m hoping this settles things, but—knowing people—I doubt it. Two guys from Florida are planning on suing Apple in a class action suit, which will like go nowhere, and the Congress—as if it didn’t have actual problems to worry about in this country at the moment—is planning on holding hearings on this weighty, troublesome matter.

It turns out, not quite relatedly, that Facebook has been putting tracking cookies onto the computers of people who aren’t themselves Facebook users, but have simply visited sites that use Facebook Connect. Facebook’s explanation? “Oops.”

iPhone SpyPhone? The Video!

24 Apr

I had a realization yesterday that, since lots of locations—as many as 43 or more at a time—get given the same time stamp in the consolidated.db file, there wasn’t any particular reason to bucket things by week, or by day. You can get a sequential picture of what the phone was doing by simply treating each set of locations as an “incident”.

I’ve put an updated version of the file from Pete Warden’s project, iPhoneTracker, onto my site for you to download if you like. It’s the version used to produce this video, which incorporates the change described, as well as increasing the resolution forty times over Pete’s version. Download my version of iPhoneTrackerAppDelegate.m from here. Alternatively, I’ve also put up a pre-built version of the iPhoneTracker application which displays the locations in your data the way I did mine for the video. (Without the soundtrack, sorry.)

A high-quality (SD) version of the video can be downloaded from here.

For those interested in the nitty-gritty, video screen capture was done with SnapZ Pro, editing with Final Cut Pro, and titling with Motion. The sound track is “My Fair Lady” by David Byrne, which was released under a Creative Commons license as part of, The Wired CD a “free music” compilation by Wired magazine several years ago. You can download all the tracks from here.

Enjoy it! I’ve had fun with it, but I’m guessing I’ve pretty much plumbed the depths of this at this point…

What do you think after seeing this video? Are you more, or less, concerned about your iPhone, or other cell phone, “keeping track of your every move”?

And check out my previous postings on this subject: iPhones and Location—Let’s Not Get Hysterical; A Follow-up on the iPhone Location Fracas; and An Even Deeper Dive Into the iPhone Location Data.


Get on my mailing list, and get the really interesting news first!

An Even Deeper Dive Into the iPhone Location Data

22 Apr

The iPhone "CellLocation" table, 6/23/10 to 6/25/10

[UPDATE: Check out the latest installment, “iPhone/SpyPhone?—The Music Video!“, with a soundtrack by David Byrne. You can also download a higher-res version of the video, my code changes for Pete Warden’s iPhoneTracker, a prebuilt version of iPhoneTracker like the one I used to make the video, and a complete copy of The Wired CD, with a bunch of great, Creative Commons-licensed tunes from David Byrne, Spoon, Gilberto Gil, the Beastie Boys and many others for you to rip, reuse and remix! Read this post now, or weep tears the size of October cabbages later!]

So, I went through the exercise of pulling an old consolidated.db table out of a backup—this seems to be the very first one from my iPhone 4. What you see above is (most of) the data plotted onto Google maps—there are a couple of outliers to the south that fell off the bottom, but they’re included in the detail maps below. This represents 3 days worth of data, which I’m nominally referring to as 6/23/10 through 6/25/10 (I’m not sure about these time stamps; more on this in a bit).

The Data

Altogether, there are 302 data points. On the map above, all the locations collected the first day are yellow, the second day’s are purple, and the third day’s are cyan.

I’ve provided a number of detail maps below, one corresponding to each unique timestamp in the table. In two cases, the locations with a given time covered an extremely wide area; in both instances, I’ve provided a detail map to show the main cluster of locations in better detail. The locations were apparently collected, or added, or something, in several batches, all with exactly the same time stamp, as follows:

6/23/10 03:55—8 locations

Eighteen hours, thirty minutes later…

6/23/10 22:25—21 locations

Sixteen hours, twenty minutes later…

6/24/10 14:45—15 locations

Eight minutes later….

6/24/10 14:53—21 locations

Seven minutes later…

6/24/10 15:00—42 locations

Five minutes later….

6/24/10 15:05—19 locations

One minute later…

6/24/10 15:06—24 locations

6/24/10 15:06—detail

Nine minutes later…

6/24/10 15:15—43 locations

Five minutes later…

6/24/10 15:20—7 locations

Eight hours, fifty-five minutes later…

6/25/10 00:15—21 locations

Four hours, thirty-seven minutes later…

6/25/10 04:52—19 locations

One minute later…

6/25/10 04:53—21 locations

Four minutes later…

6/25/10 04:57—41 locations

6/25/10 04:57—detail

Some Oddities

Now, about the timestamps. I’ve used the same conversion that Peter Warden uses in his iPhoneTracker application, however, the first set of dates comes out as June 23rd. The iPhone 4 was released June 24th, and I picked mine up that evening, so it seems that something is amiss here.

The way the locations are clustered by time stamp seem bizarre, at least if they’re meant to indicate the time the entry was created. No multiday gaps in here, but multi-hour ones and an extremely wide variance in both the number of locations with a given time stamp and the geographical range encompassing those locations. In particular, the locations with the time stamp of 15:06 on the 24th and 04:57 on the 25th cover a great deal of ground. Why locations would be added in this fashion, and why those locations at any given moment is a mystery.

Some Observations

The fact that anywhere from 7 to 43 locations have the same time stamp, that those locations can demonstrably be spread over an area of ninety or a hundred miles (as in the case of the latest-dated set of locations) and the wide variance in the time between subsequent time stamps makes the idea that the iPhone is “tracking your every move” or even very many of them, pretty questionable.

“Tracking your every move” is what my geo-tagger does: once it gets a fix from GPS, it makes a note of where I am every five seconds, if at all possible. That allowed me to put together tracks like the ones I show on the web pages on my pilgrimage in Japan in 2009, reproduced below. This is not what the iPhone is doing.

I’d like to try to verify at some point whether the locations found in consolidated.db are, indeed, cell towers, as I suspect. In some cases, it seems as though there might be too many of them in too small an area, but there may be some other explanation for that.

How I Did This

First, I had to find the consolidated.db file. Anyone who tells you this is easy has never done it. First, you have to find the appropriate backup folder in ~/Library/ApplicationSupport/MobileSync/Backups/ folder. The folder in the Backups directory have names like “8a0bf15905f9cb6e3e6df6fc551467676f3873dc”, and there may be several for every device you sync to the desktop system. To determine the correct one, you have to go into the folder and examine the file “Info.plist” and look for the device name corresponding to your iPhone.

Once you’ve accomplished that, you have to determine which of the files in there is actually the consolidated.db database file, since all the files also have names like “fe5632d8f9e0966173c59da3d92af864bb8cfdc6”. To do this, you’re going to need to decipher the manifest, which is in the files Manifest.dmdb and Manifest.dmdx. A Python script developed by the user “galloglass” in response to a question on StackOverflow accomplishes this. (You can download a copy of this script from my website.) Copy the two Manifest files to the same directory as the script, and the command

./ParseManifest.py | grep consolidated

should produce some output along the lines of

-rw-r--r-- 00000000 00000000  528384 1277443751 1277443751 1277166641
   (4096c9ec676f2847dc283405900e284a7c815836)
     RootDomain::Library/Caches/locationd/
       consolidated.db

The file name is the long, incomprehensible string in the parentheses. Copy the file with that name from the backup folder to your work directory, and rename it “consolidated.db”.

Next, you’ll need to dump the “CellLocations” table from the file. You can use any SQLite-capable tool you like for this. I used RazorSQL, which is a decent enough OS X application to manage this, and lets you dump an entire table to an Excel spreadsheet, or a variety of other formats easily. (It’s a free trial for 30 days.) Once you’ve done that, you’re good to go.

(I’ve uploaded the Excel spreadsheet I used to do this analysis, so you can download a copy to play with as well. The only addition I’ve made is to add a column to the right with the date corresponding (according to Peter’s kind of suspect conversion) to the time stamp. You can get a copy from here.)

Given a spreadsheet, putting the data onto a Google Map takes a little work, but nothing insurmountable. I used the Google Earth Spreadsheet Mapper, but I’m not proposing to provide a tutorial here.

If people come up with any exciting discoveries playing around with this, I’d be interested in hearing about them. My guess is still that this is most likely cell tower data for use as a cache for triangulation. Where it comes from, exactly, I have no actual idea, but I’d guess that the iPhone makes a note of any cell tower it becomes aware of, regardless of the signal strength, and obtains its geographical location from Apple by some means. I’ll likewise be interested in hearing what kind of response Apple comes up with to the questions raised by Sen. Franken and others.

But me, I’m not especially worried about the existence of this data on my phone.

A Follow-up on the iPhone Location Fracas

22 Apr

[UPDATE: Check out the latest installment, “iPhone/SpyPhone?—The Music Video!“, with a soundtrack by David Byrne. You can also download a higher-res version of the video, my code changes for Pete Warden’s iPhoneTracker, a prebuilt version of iPhoneTracker like the one I used to make the video, and a complete copy of The Wired CD, with a bunch of great, Creative Commons-licensed tunes from David Byrne, Spoon, Gilberto Gil, the Beastie Boys and many others for you to rip, reuse and remix! Read this post now, or weep tears the size of October cabbages later!]

[UPDATE: I’ve taken a closer look at the data by dumping out the database’s CellLocation table to a spreadsheet, and mapping it over time. I’ve made a copy of the spreadsheet with the data available for download. See here for details.]

My posting on my investigations into the content of the consolidated.db file on the iPhone has gotten some 40,000 views, so far, thanks to the magic of Slashdot. There have been a couple of worthwhile items that came up in comments, and I wanted to collect them into a follow-up posting here.

First, Alex Levinson, a researcher who’s done academic work on iOS forensics, posted an excellent column on this which probably deserved to make it to Slashdot more than mine did. It turns out that the existence of this file was not only known, but mentioned in Sean Morrisey’s book on iOS forensics, to which Alex was a contributor.

Second, even without a handy database of what is increasingly appearing to be cell tower and WiFi hotspot locations, people should be aware that their cell phone—as a simple consequence of its operation—”tracks” their movements, simply to enable the hand-off of the phone from one cel tower to the next. Your carrier maintains this information for some period of time, and will provide it to law enforcement in response to an appropriate subpoena.

Interestingly, a German politician, Malte Spitz, sued his carrier, Deutsche Telekomm, to get a copy of the records that they had maintained on him, and discovered that, between August 2009 and February 2010, they had recorded his geographical location some 35,000 times. Zeit Online has a fascinating visualization of Mr. Spitz’s movement and activities developed from this data.

Finally, and sadly, Brian Chen over at Wired has a follow-on to his original column where he gets off to a bad start by noting that people had been “spooked” by the revelation of the existence of this file on their iPhones, but without noting that it was his own headline the previous day—which claimed that iPhones were “tracking [their owners’] every move”, inaccurately as it turns out—which engendered a lot of the “spooking”.

If you’re concerned about this file’s being backed up to your desktop, I’d recommend that you turn on encrypted backups, which can be accomplished through iTunes, as this posting on Techland explains. I still haven’t got the slightest idea why people would be particularly worried about thieves getting this particular file off their desktops, but not (apparently) concerned about their address books, their email archives, their document folders or their calendars.

People are strange.