Which day of the week?

Hurricane Andrew, which hit Florida on August 24, 1992, was one of the costliest and deadliest hurricanes in US history. Which day of the week did it make landfall?

Let’s walk through all of the steps to figure this out.

# Import date from datetime
from datetime import date

# Create a date object
hurricane_andrew = date(1992, 8, 24)

# Which day of the week is the date?
print(hurricane_andrew.weekday())

<script.py> output:
    0

How many hurricanes come early?

In this chapter, you will work with a list of the hurricanes that made landfall in Florida from 1950 to 2017. There were 235 in total. Check out the variable florida_hurricane_dates, which has all of these dates.

Atlantic hurricane season officially begins on June 1. How many hurricanes since 1950 have made landfall in Florida before the official start of hurricane season?

Instructions

• Complete the for loop to iterate through florida_hurricane_dates. 
• Complete the if statement to increment the counter (early_hurricanes) if the hurricane made landfall before June.

# Counter for how many before June 1
early_hurricanes = 0

# We loop over the dates
for hurricane in florida_hurricane_dates:
  # Check if the month is before June (month number 6)
  if hurricane.month < 6:
    early_hurricanes = early_hurricanes + 1
    
print(early_hurricanes)

10

Subtracting dates

Python date objects let us treat calendar dates as something similar to numbers: we can compare them, sort them, add, and even subtract them. This lets us do math with dates in a way that would be a pain to do by hand.

The 2007 Florida hurricane season was one of the busiest on record, with 8 hurricanes in one year. The first one hit on May 9th, 2007, and the last one hit on December 13th, 2007. How many days elapsed between the first and last hurricane in 2007?

Instructions

• Import date from datetime. 
• Create a date object for May 9th, 2007, and assign it to the start variable.
• Create a date object for December 13th, 2007, and assign it to the end variable.
• Subtract start from end, to print the number of days in the resulting timedelta object.

# Import date
from datetime import date

# Create a date object for May 9th, 2007
start = date(2007, 5, 9)

# Create a date object for December 13th, 2007
end = date(2007, 12, 13)

# Subtract the two dates and print the number of days
print((end - start).days)

<script.py> output:
    218

One thing to note: be careful using this technique for historical dates hundreds of years in the past. Our calendar systems have changed over time, and not every date from then would be the same day and month today.

Counting events per calendar month

Hurricanes can make landfall in Florida throughout the year. As we’ve already discussed, some months are more hurricane-prone than others.

Using florida_hurricane_dates, let’s see how hurricanes in Florida were distributed across months throughout the year. 

We’ve created a dictionary called hurricanes_each_month to hold your counts and set the initial counts to zero. You will loop over the list of hurricanes, incrementing the correct month in hurricanes_each_month as you go, and then print the result.

Instructions

• Within the for loop: 
• Assign month to be the month of that hurricane. 
• Increment hurricanes_each_month for the relevant month by 1.

# A dictionary to count hurricanes per calendar month
hurricanes_each_month = {1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6:0,
		  				 7: 0, 8:0, 9:0, 10:0, 11:0, 12:0}

# Loop over all hurricanes
for hurricane in florida_hurricane_dates:
  # Pull out the month
  month = hurricane.month
  # Increment the count in your dictionary by one
  hurricanes_each_month[month] = hurricanes_each_month[month] + 1
  
print(hurricanes_each_month)

<script.py> output:
    {1: 0, 2: 1, 3: 0, 4: 1, 5: 8, 6: 32, 7: 21, 8: 49, 9: 70, 10: 43, 11: 9, 12: 1}

This illustrated a generally useful pattern for working with complex data: creating a dictionary, performing some operation on each element, and storing the results back in the dictionary.

Putting a list of dates in order

Much like numbers and strings, date objects in Python can be put in order. Earlier dates come before later ones, and so we can sort a list of date objects from earliest to latest.

What if our Florida hurricane dates had been scrambled? We’ve gone ahead and shuffled them so they’re in random order and saved the results as dates_scrambled. Your job is to put them back in chronological order, and then print the first and last dates from this sorted list.

Instructions 

• Sort dates_scrambled using Python’s built-in sorted()method, and save the results to dates_ordered. 
• Print the first and last dates in dates_ordered.

# Print the first and last scrambled dates
print(dates_scrambled[0])
print(dates_scrambled[-1])

# Put the dates in order
dates_ordered = sorted(dates_scrambled)

# Print the first and last ordered dates
print(dates_ordered[0])
print(dates_ordered[-1])

1988-08-04
2011-07-18
1950-08-31
2017-10-29

You can use sorted() on several data types in Python, including sorting lists of numbers, lists of strings, or even lists of lists, which by default are compared on the first element.

Printing dates in a friendly format

Because people may want to see dates in many different formats, Python comes with very flexible functions for turning date objects into strings.

Let’s see what event was recorded first in the Florida hurricane data set. In this exercise, you will format the earliest date in the florida_hurriance_dates list in two ways so you can decide which one you want to use: either the ISO standard or the typical US style.

Instructions

• Assign the earliest date in florida_hurricane_dates to first_date.
• Print first_date in the ISO standard. For example, December 1st, 2000 would be "2000-12-01".
• Print first_date in the US style, using .strftime(). For example, December 1st, 2000 would be “12/1/2000” .

# Assign the earliest date to first_date
first_date = min(florida_hurricane_dates)

# Convert to ISO and US formats
iso = "Our earliest hurricane date: " + first_date.isoformat()
us = "Our earliest hurricane date: " + first_date.strftime("%m/%d/%Y")

print("ISO: " + iso)
print("US: " + us)

<script.py> output:
    ISO: Our earliest hurricane date: 1950-08-31
    US: Our earliest hurricane date: 08/31/1950

When in doubt, use the ISO-8601 format for dates. ISO dates are unambiguous. And if you sort them ‘alphabetically’, for example, in filenames, they will be in the correct order.

Representing dates in different ways

date objects in Python have a great number of ways they can be printed out as strings. In some cases, you want to know the date in a clear, language-agnostic format. In other cases, you want something which can fit into a paragraph and flow naturally.

Let’s try printing out the same date, August 26, 1992 (the day that Hurricane Andrew made landfall in Florida), in a number of different ways, to practice using the .strftime() method.

A date object called andrew has already been created.

Instructions 

• Print andrew in the format ‘YYYY-MM’.
• Print andrew in the format ‘MONTH (YYYY)’, using %B for the month’s full name, which in this case will be August.
• Print andrew in the format ‘YYYY-DDD’ (where DDD is the day of the year) using %j.

# Import date
from datetime import date

# Create a date object
andrew = date(1992, 8, 26)

# Print andrew in the format 'YYYY-MM'.
print(andrew.strftime('%Y-%m'))

# Print andrew in the format 'MONTH (YYYY)', 
# using %B for the month's full name, 
# which in this case will be August.
print(andrew.strftime('%B (%Y)'))


# Print the date in the format 'YYYY-DDD'
print(andrew.strftime('%Y-%j'))

Pick the format that best matches your needs. For example, astronomers usually use the ‘day number’ out of 366 instead of the month and date, to avoid ambiguities between languages.

Creating datetimes by hand

Often you create datetime objects based on outside data. Sometimes though, you want to create a datetime object from scratch.

You’re going to create a few different datetime objects from scratch to get the hang of that process. These come from the bikeshare data set that you’ll use throughout the rest of the chapter.

Instructions

• 1
  • Import the datetime class.
  • Create a datetime for October 1, 2017 at 15:26:26.
  • Print the results in ISO format.
• 2
  • Import the datetime class.
  • Create a datetime for December 31, 2017 at 15:19:13.
  • Print the results in ISO format.

# Import datetime
from datetime import datetime

# Create a datetime object
dt = datetime(2017, 10, 1, 15, 26, 26)

# Print the results in ISO 8601 format
print(dt.isoformat())

# Create a datetime object
dt2 = datetime(2017, 12, 31,15,19,13)

# Print the results in ISO 8601 format
print(dt2.isoformat())

# Replace the year with 1917
dt_old = dt2.replace(year=1917)

# Print the results in ISO 8601 format
print(dt_old)

<script.py> output:
    2017-10-01T15:26:26

<script.py> output:
    2017-12-31T15:19:13

<script.py> output:
    1917-12-31 15:19:13

Counting events before and after noon

In this chapter, you will be working with a list of all bike trips for one Capital Bikeshare bike, W20529, from October 1, 2017 to December 31, 2017. This list has been loaded as onebike_datetimes. 

Each element of the list is a dictionary with two entries: start is a datetime object corresponding to the start of a trip (when a bike is removed from the dock) and end is a datetime object corresponding to the end of a trip (when a bike is put back into a dock).

You can use this data set to understand better how this bike was used. Did more trips start before noon or after noon?

Instructions

• Within the for loop, complete the if statement to check if the trip started before noon.
• Within the for loop, increment trip_counts['AM'] if the trip started before noon, and trip_counts['PM'] if it started after noon.

# Create dictionary to hold results
trip_counts = {'AM': 0, 'PM': 0}
  
# Loop over all trips
for trip in onebike_datetimes:
  # Check to see if the trip starts before noon
  if trip['start'].hour < 12:
    # Increment the counter for before noon
    trip_counts['AM'] += 1
  else:
    # Increment the counter for after noon
    trip_counts['PM'] += 1
  
print(trip_counts)

<script.py> output:
    {'AM': 94, 'PM': 196}

It looks like this bike is used about twice as much after noon than it is before noon. One obvious follow up would be to see which hours the bike is most likely to be taken out for a ride.

Turning strings into datetimes

When you download data from the Internet, dates and times usually come to you as strings. Often the first step is to turn those strings into datetime objects.

In this exercise, you will practice this transformation.

Instructions 1/3

• 1
  • Determine the format needed to convert s to datetime and assign it to fmt. 
  • Convert the string s to datetime using fmt.

# Import the datetime class
from datetime import datetime

# Starting string, in YYYY-MM-DD HH:MM:SS format
s = '2017-02-03 00:00:01'

# Write a format string to parse s
fmt = '%Y-%m-%d %H:%M:%S'

# Create a datetime object d
d = datetime.strptime(s, fmt)

# Print d
print(d)

2017-02-03 00:00:01

structions 2/3

• 2
  • Determine the format needed to convert s to datetime and assign it to fmt. 
  • Convert the string s to datetime using fmt.

# Import the datetime class
from datetime import datetime

# Starting string, in YYYY-MM-DD format
s = '2030-10-15'

# Write a format string to parse s
fmt = '%Y-%m-%d'

# Create a datetime object d
d = datetime.strptime(s, fmt)

# Print d
print(d)

<script.py> output:
    2030-10-15 00:00:00

Instructions 3/3

• 3
  • Determine the format needed to convert s to datetime and assign it to fmt. 
  • Convert the string s to datetime using fmt.

# Import the datetime class
from datetime import datetime

# Starting string, in MM/DD/YYYY HH:MM:SS format
s = '12/15/1986 08:00:00'

# Write a format string to parse s
fmt = '%m/%d/%Y %H:%M:%S'

# Create a datetime object d
d = datetime.strptime(s, fmt)

# Print d
print(d)

<script.py> output:
    1986-12-15 08:00:00

Now you can parse dates in most common formats. Unfortunately, Python does not have the ability to parse non-zero-padded dates and times out of the box (such as 1/2/2018). If needed, you can use other string methods to create zero-padded strings suitable for strptime().

Parsing pairs of strings as datetimes

Up until now, you’ve been working with a pre-processed list of datetimes for W20529’s trips. For this exercise, you’re going to go one step back in the data cleaning pipeline and work with the strings that the data started as. 

Explore onebike_datetime_strings in the IPython shell to determine the correct format. datetime has already been loaded for you.

Instructions

• Outside the for loop, fill out the fmt string with the correct parsing format for the data.
• Within the for loop, parse the start and end strings into the trip dictionary with start and end keys and datetime objects for values.

# Write down the format string
fmt = "%Y-%m-%d %H:%M:%S"

# Initialize a list for holding the pairs of datetime objects
onebike_datetimes = []

# Loop over all trips
for (start, end) in onebike_datetime_strings:
  trip = {'start': datetime.strptime(start, fmt),
          'end': datetime.strptime(end, fmt)}
  
  # Append the trip
  onebike_datetimes.append(trip)

Now you know how to process lists of strings into a more useful structure. If you haven’t come across this approach before, many complex data cleaning tasks follow this same format: start with a list, process each element, and add the processed data to a new list.

Recreating ISO format with strftime()

In the last chapter, you used strftime() to create strings from date objects. Now that you know about datetime objects, let’s practice doing something similar.

Re-create the .isoformat() method, using .strftime(), and print the first trip start in our data set. 

Instructions

• Complete fmt to match the format of ISO 8601.
• Print first_start with both .isoformat() and .strftime(); they should match.

# Import datetime
from datetime import datetime

# Pull out the start of the first trip
first_start = onebike_datetimes[0]['start']

# Format to feed to strftime()
fmt = "%Y-%m-%dT%H:%M:%S"

# Print out date with .isoformat(), then with .strftime() to compare
print(first_start.isoformat())
print(first_start.strftime(fmt))

2017-10-01T15:23:25
2017-10-01T15:23:25

There are a wide variety of time formats you can create with strftime(), depending on your needs. However, if you don’t know exactly what you need, .isoformat() is a perfectly fine place to start.

Unix timestamps

Datetimes are sometimes stored as Unix timestamps: the number of seconds since January 1, 1970. This is especially common with computer infrastructure, like the log files that websites keep when they get visitors.

Instructions

• Complete the for loop to loop over timestamps.
• Complete the code to turn each timestamp ts into a datetime.

# Import datetime
from datetime import datetime

# Starting timestamps
timestamps = [1514665153, 1514664543]

# Datetime objects
dts = []

# Loop
for ts in timestamps:
  dts.append(datetime.fromtimestamp(ts))
  
# Print results
print(dts)

[datetime.datetime(2017, 12, 30, 20, 19, 13), datetime.datetime(2017, 12, 30, 20, 9, 3)]

The largest number that some older computers can hold in one variable is 2147483648, which as a Unix timestamp is in January 2038. On that day, many computers which haven’t been upgraded will fail. Hopefully, none of them are running anything critical!

Turning pairs of datetimes into durations

When working with timestamps, we often want to know how much time has elapsed between events. Thankfully, we can use datetimearithmetic to ask Python to do the heavy lifting for us so we don’t need to worry about day, month, or year boundaries. Let’s calculate the number of seconds that the bike was out of the dock for each trip.

Continuing our work from a previous coding exercise, the bike trip data has been loaded as the list onebike_datetimes. Each element of the list consists of two datetime objects, corresponding to the start and end of a trip, respectively.

Instructions

• Within the loop:
  • Use arithmetic on the start and end elements to find the length of the trip
  • Save the results to trip_duration.
  • Calculate trip_length_seconds from trip_duration.

# Initialize a list for all the trip durations
onebike_durations = []

for trip in onebike_datetimes:
  # Create a timedelta object corresponding to the length of the trip
  trip_duration = trip['end'] - trip['start']
  
  # Get the total elapsed seconds in trip_duration
  trip_length_seconds = trip_duration.total_seconds()
  
  # Append the results to our list
  onebike_durations.append(trip_length_seconds)

[181.0, 7622.0, 343.0, 1278.0, 1277.0, 1366.0, 815.0, 545.0, 491.0, 639.0, 1678.0, 406.0, 709.0, 514.0, 492.0, 1668.0, 2242.0, 2752.0, 735.0, 330.0, 518.0, 1433.0, 204.0, 304.0, 977.0, 1399.0, 1244.0, 658.0, 800.0, 1911.0, 2471.0, 1344.0, 435.0, 271.0, 920.0, 851.0, 209.0, 453.0, 841.0, 142.0, 1023.0, 1466.0, 1636.0, 3039.0, 1571.0, 1410.0, 386.0, 1527.0, 622.0, 1450.0, 1422.0, 991.0, 1484.0, 1450.0, 929.0, 533.0, 525.0, 283.0, 133.0, 1106.0, 952.0, 553.0, 659.0, 297.0, 357.0, 989.0, 979.0, 760.0, 1110.0, 675.0, 1207.0, 1593.0, 768.0, 1446.0, 485.0, 200.0, 399.0, 242.0, 170.0, 450.0, 1078.0, 1042.0, 573.0, 748.0, 735.0, 336.0, 76913.0, 171.0, 568.0, 358.0, 917.0, 671.0, 1791.0, 318.0, 888.0, 1284.0, 11338.0, 1686.0, 5579.0, 8290.0, 1850.0, 1810.0, 870.0, 436.0, 429.0, 494.0, 1439.0, 380.0, 629.0, 962.0, 387.0, 952.0, 190.0, 739.0, 1120.0, 369.0, 2275.0, 873.0, 1670.0, 643.0, 572.0, 1375.0, 725.0, 688.0, 1041.0, 1707.0, 1236.0, 1291.0, 2890.0, -3346.0, 1213.0, 331.0, 1497.0, 527.0, 584.0, 2599.0, 759.0, 1291.0, 916.0, 161.0, 806.0, 838.0, 644.0, 374.0, 678.0, 137.0, 659.0, 386.0, 745.0, 448.0, 558.0, 888.0, 662.0, 663.0, 362.0, 513.0, 655.0, 221.0, 469.0, 430.0, 192.0, 324.0, 1233.0, 923.0, 961.0, 525.0, 1017.0, 1216.0, 747.0, 668.0, 1219.0, 1182.0, 10262.0, 1106.0, 399.0, 724.0, 330.0, 499.0, 968.0, 1310.0, 2629.0, 427.0, 839.0, 258.0, 396.0, 238.0, 745.0, 613.0, 710.0, 2068.0, 947.0, 1509.0, 254.0, 625.0, 479.0, 688.0, 238.0, 322.0, 304.0, 576.0, 1035.0, 661.0, 276.0, 1427.0, 998.0, 729.0, 723.0, 220.0, 212.0, 759.0, 268.0, 374.0, 305.0, 304.0, 289.0, 2620.0, 1288.0, 212.0, 2656.0, 996.0, 271.0, 701.0, 458.0, 116.0, 124.0, 276.0, 532.0, 257.0, 1089.0, 195.0, 384.0, 511.0, 850.0, 462.0, 322.0, 998.0, 327.0, 153.0, 152.0, 230.0, 321.0, 625.0, 391.0, 1298.0, 1018.0, 220.0, 277.0, 221.0, 216.0, 509.0, 596.0, 367.0, 447.0, 257.0, 1049.0, 1367.0, 933.0, 151.0, 153.0, 1336.0, 298.0, 380.0, 1385.0, 225.0, 181.0, 581.0, 250.0, 332.0, 845.0, 277.0, 663.0, 541.0, 652.0, 257.0, 775.0, 499.0, 1744.0, 445.0, 297.0, 1716.0, 2046.0, 809.0, 1287.0, 596.0, 1513.0, 651.0, 625.0, 279.0, 210.0, 610.0]

Remember that timedelta objects are represented in Python as a number of days and seconds of elapsed time. Be careful not to use .seconds on a timedelta object, since you’ll just get the number of seconds without the days!

Average trip time

W20529 took 291 trips in our data set. How long were the trips on average? We can use the built-in Python functions sum() and len() to make this calculation.

Based on your last coding exercise, the data has been loaded as onebike_durations. Each entry is a number of seconds that the bike was out of the dock.

Instructions

• Calculate total_elapsed_time across all trips in onebike_durations.
• Calculate number_of_trips for onebike_durations.
• Divide total_elapsed_time by number_of_trips to get the average trip length.

# What was the total duration of all trips?
total_elapsed_time = sum(onebike_durations)

# What was the total number of trips?
number_of_trips = len(onebike_durations)
  
# Divide the total duration by the number of trips
print(total_elapsed_time / number_of_trips)

1178.9310344827586

For the average to be a helpful summary of the data, we need for all of our durations to be reasonable numbers, and not a few that are way too big, way too small, or even malformed. For example, if there is anything fishy happening in the data, and our trip ended before it started, we’d have a negative trip length.

The long and the short of why time is hard

Out of 291 trips taken by W20529, how long was the longest? How short was the shortest? Does anything look fishy?

As before, data has been loaded as onebike_durations.

Instructions

• Calculate shortest_trip from onebike_durations.
• Calculate longest_trip from onebike_durations.
• Print the results, turning shortest_trip and longest_trip into strings so they can print.

# Calculate shortest and longest trips
shortest_trip = min(onebike_durations)
longest_trip = max(onebike_durations)

# Print out the results
print("The shortest trip was " + str(shortest_trip) + " seconds")
print("The longest trip was " + str(longest_trip) + " seconds")

<script.py> output:
    The shortest trip was -3346.0 seconds
    The longest trip was 76913.0 seconds

For at least one trip, the bike returned before it left. Why could that be? Here’s a hint: it happened in early November, around 2AM local time. What happens to clocks around that time each year? By the end of the next chapter, we’ll have all the tools we need to deal with this situation!

Creating timezone aware datetimes

In this exercise, you will practice setting timezones manually.

Instructions 1/3

• 1
  • Import timezone. 
  • Set the tzinfo to UTC, without using timedelta.Take Hint (-10 XP)

# Import datetime, timezone
from datetime import datetime, timezone

# October 1, 2017 at 15:26:26, UTC
dt = datetime(2017, 10, 1, 15, 26, 26, tzinfo=timezone.utc)

# Print results
print(dt.isoformat())

<script.py> output:
    2017-10-01T15:26:26+00:00

• 
  2
  • Set pst to be a timezone set for UTC-8.
  • Set dt‘s timezone to be pst.

# Import datetime, timedelta, timezone
from datetime import datetime, timedelta, timezone

# Create a timezone for Pacific Standard Time, or UTC-8
pst = timezone(timedelta(hours=-8))

# October 1, 2017 at 15:26:26, UTC-8
dt = datetime(2017, 10, 1, 15, 26, 26, tzinfo=pst)

# Print results
print(dt.isoformat())

<script.py> output:
    2017-10-01T15:26:26-08:00

• 
  3
  • Set tz to be a timezone set for UTC+11.
  • Set dt‘s timezone to be tz.

from datetime import datetime, timedelta, timezone

# Create a timezone for Australian Eastern Daylight Time, or UTC+11
aedt = timezone(timedelta(hours=11))

# October 1, 2017 at 15:26:26, UTC+11
dt = datetime(2017, 10, 1, 15, 26, 26, tzinfo=aedt)

# Print results
print(dt.isoformat())

2017-10-01T15:26:26+11:00

Did you know that Russia and France are tied for the most number of time zones, with 12 each? The French mainland only has one timezone, but because France has so many overseas dependencies they really add up!

Setting timezones

Now that you have the hang of setting timezones one at a time, let’s look at setting them for the first ten trips that W20529 took.

timezone and timedelta have already been imported. Make the change using .replace()

Instructions

• Create edt, a timezone object whose UTC offset is -4 hours.
• Within the for loop:
• Set the tzinfo for trip['start'].
• Set the tzinfo for trip['end'].

edt = timezone(timedelta(hours=-4))

# Loop over trips, updating the start and end datetimes to be in UTC-4
for trip in onebike_datetimes[:10]:
  # Update trip['start'] and trip['end']
  trip['start'] = trip['start'].replace(tzinfo=edt)
  trip['end'] = trip['end'].replace(tzinfo=edt)

Did you know that despite being over 2,500 miles (4,200 km) wide (about as wide as the continential United States or the European Union) China has only one official timezone? There’s a second, unofficial timezone, too. It is used by much of the Uyghurs population in the Xinjiang province in the far west of China.

What time did the bike leave in UTC?

Having set the timezone for the first ten rides that W20529 took, let’s see what time the bike left in UTC. We’ve already loaded the results of the previous exercise into memory.

Instructions

Within the for loop, move dt to be in UTC. Use timezone.utc as a convenient shortcut for UTC.

# Loop over the trips
for trip in onebike_datetimes[:10]:
  # Pull out the start
  dt = trip['start']
  # Move dt to be in UTC
  dt = dt.astimezone(timezone.utc)
  
  # Print the start time in UTC
  print('Original:', trip['start'], '| UTC:', dt.isoformat())

<script.py> output:
    Original: 2017-10-01 15:23:25-04:00 | UTC: 2017-10-01T19:23:25+00:00
    Original: 2017-10-01 15:42:57-04:00 | UTC: 2017-10-01T19:42:57+00:00
    Original: 2017-10-02 06:37:10-04:00 | UTC: 2017-10-02T10:37:10+00:00
    Original: 2017-10-02 08:56:45-04:00 | UTC: 2017-10-02T12:56:45+00:00
    Original: 2017-10-02 18:23:48-04:00 | UTC: 2017-10-02T22:23:48+00:00
    Original: 2017-10-02 18:48:08-04:00 | UTC: 2017-10-02T22:48:08+00:00
    Original: 2017-10-02 19:18:10-04:00 | UTC: 2017-10-02T23:18:10+00:00
    Original: 2017-10-02 19:37:32-04:00 | UTC: 2017-10-02T23:37:32+00:00
    Original: 2017-10-03 08:24:16-04:00 | UTC: 2017-10-03T12:24:16+00:00
    Original: 2017-10-03 18:17:07-04:00 | UTC: 2017-10-03T22:17:07+00:00

Did you know that there is no official time zone at the North or South pole? Since all the lines of longitude meet each other, it’s up to each traveler (or research station) to decide what time they want to use.

Putting the bike trips into the right time zone

Instead of setting the timezones for W20529 by hand, let’s assign them to their IANA timezone: ‘America/New_York’. Since we know their political jurisdiction, we don’t need to look up their UTC offset. Python will do that for us.

Instructions

• Import tz from dateutil.
• Assign et to be the timezone 'America/New_York'.
• Within the for loop, set start and end to have et as their timezone (use .replace()).

# Import tz
from dateutil import tz

# Create a timezone object for Eastern Time
et = tz.gettz('America/New_York')

# Loop over trips, updating the datetimes to be in Eastern Time
for trip in onebike_datetimes[:10]:
  # Update trip['start'] and trip['end']
  trip['start'] = trip['start'].replace(tzinfo=et)
  trip['end'] = trip['end'].replace(tzinfo=et)

Time zone rules actually change quite frequently. IANA time zone data gets updated every 3-4 months, as different jurisdictions make changes to their laws about time or as more historical information about timezones are uncovered. tz is smart enough to use the date in your datetime to determine which rules to use historically.

What time did the bike leave? (Global edition)

When you need to move a datetime from one timezone into another, use .astimezone() and tz. Often you will be moving things into UTC, but for fun let’s try moving things from ‘America/New_York’ into a few different time zones.

Instructions 1/3

• 1
  • Set uk to be the timezone for the UK: ‘Europe/London’.
  • Change local to be in the uk timezone and assign it to notlocal.

uk = tz.gettz('Europe/London')

# Pull out the start of the first trip
local = onebike_datetimes[0]['start']

# What time was it in the UK?
notlocal = local.astimezone(uk)

# Print them out and see the difference
print(local.isoformat())
print(notlocal.isoformat())

<script.py> output:
    2017-10-01T15:23:25-04:00
    2017-10-01T20:23:25+01:00

• 
  2
  • Set ist to be the timezone for India: ‘Asia/Kolkata’.
  • Change local to be in the ist timezone and assign it to notlocal.

# Create the timezone object
ist = tz.gettz('Asia/Kolkata')

# Pull out the start of the first trip
local = onebike_datetimes[0]['start']

# What time was it in India?
notlocal = local.astimezone(ist)

# Print them out and see the difference
print(local.isoformat())
print(notlocal.isoformat())

<script.py> output:
    2017-10-01T15:23:25-04:00
    2017-10-02T00:53:25+05:30

• 
  3
  • Set sm to be the timezone for Samoa: ‘Pacific/Apia’.
  • Change local to be in the sm timezone and assign it to notlocal.

# Create the timezone object
sm = tz.gettz('Pacific/Apia')

# Pull out the start of the first trip
local = onebike_datetimes[0]['start']

# What time was it in Samoa?
notlocal = local.astimezone(sm)

# Print them out and see the difference
print(local.isoformat())
print(notlocal.isoformat())

<?php<script.py> output:
    2017-10-01T15:23:25-04:00
    2017-10-02T09:23:25+14:00

Did you notice the time offset for this one? It’s at UTC+14! Samoa used to be UTC-10, but in 2011 it changed to the other side of the International Date Line to better match New Zealand, its closest trading partner. However, they wanted to keep the clocks the same, so the UTC offset shifted from -10 to +14, since 24-10 is 14. Timezones… not simple!

How many hours elapsed around daylight saving?

Since our bike data takes place in the fall, you’ll have to do something else to learn about the start of daylight savings time.

Let’s look at March 12, 2017, in the Eastern United States, when Daylight Saving kicked in at 2 AM.

If you create a datetime for midnight that night, and add 6 hours to it, how much time will have elapsed?

Instructions 1/3

• 1

You already have a datetime called start, set for March 12, 2017 at midnight, set to the timezone ‘America/New_York’. 

Add six hours to start and assign it to end. Look at the UTC offset for the two results.

# Import datetime, timedelta, tz, timezone
from datetime import datetime, timedelta, timezone
from dateutil import tz

# Start on March 12, 2017, midnight, then add 6 hours
start = datetime(2017, 3, 12, tzinfo = tz.gettz('America/New_York'))
end = start + timedelta(hours=6)
print(start.isoformat() + " to " + end.isoformat())

<script.py> output:
    2017-03-12T00:00:00-05:00 to 2017-03-12T06:00:00-04:00

You added 6 hours, and got 6 AM, despite the fact that the clocks springing forward means only 5 hours would have actually elapsed!

Calculate the time between start and end. How much time does Python think has elapsed?

# Import datetime, timedelta, tz, timezone
from datetime import datetime, timedelta, timezone
from dateutil import tz

# Start on March 12, 2017, midnight, then add 6 hours
start = datetime(2017, 3, 12, tzinfo = tz.gettz('America/New_York'))
end = start + timedelta(hours=6)
print(start.isoformat() + " to " + end.isoformat())

# How many hours have elapsed?
print((end - start).total_seconds()/(60*60))

<script.py> output:
    2017-03-12T00:00:00-05:00 to 2017-03-12T06:00:00-04:00
    6.0

Move your datetime objects into UTC and calculate the elapsed time again.

Once you’re in UTC, what result do you get?

# Import datetime, timedelta, tz, timezone
from datetime import datetime, timedelta, timezone
from dateutil import tz

# Start on March 12, 2017, midnight, then add 6 hours
start = datetime(2017, 3, 12, tzinfo = tz.gettz('America/New_York'))
end = start + timedelta(hours=6)
print(start.isoformat() + " to " + end.isoformat())

# How many hours have elapsed?
print((end - start).total_seconds()/(60*60))

# What if we move to UTC?
print((end.astimezone(timezone.utc) - start.astimezone(timezone.utc))\
      .total_seconds()/(60*60))

<script.py> output:
    2017-03-12T00:00:00-05:00 to 2017-03-12T06:00:00-04:00
    6.0
    5.0

When we compare times in local time zones, everything gets converted into clock time. Remember if you want to get absolute time differences, always move to UTC!

March 29, throughout a decade

Daylight Saving rules are complicated: they’re different in different places, they change over time, and they usually start on a Sunday (and so they move around the calendar).

For example, in the United Kingdom, as of the time this lesson was written, Daylight Saving begins on the last Sunday in March. Let’s look at the UTC offset for March 29, at midnight, for the years 2000 to 2010.

Instructions

• Using tz, set the timezone for dt to be 'Europe/London'.
• Within the for loop:
• Use the .replace() method to change the year for dt to be y.
• Call .isoformat() on the result to observe the results.

# Import datetime and tz
from datetime import datetime
from dateutil import tz

# Create starting date
dt = datetime(2000, 3, 29, tzinfo = tz.gettz('Europe/London'))

# Loop over the dates, replacing the year, and print the ISO timestamp
for y in range(2000, 2011):
  print(dt.replace(year=y).isoformat())

<script.py> output:
    2000-03-29T00:00:00+01:00
    2001-03-29T00:00:00+01:00
    2002-03-29T00:00:00+00:00
    2003-03-29T00:00:00+00:00
    2004-03-29T00:00:00+01:00
    2005-03-29T00:00:00+01:00
    2006-03-29T00:00:00+01:00
    2007-03-29T00:00:00+01:00
    2008-03-29T00:00:00+00:00
    2009-03-29T00:00:00+00:00
    2010-03-29T00:00:00+01:00

As you can see, the rules for Daylight Saving are not trivial. When in doubt, always use tz instead of hand-rolling timezones, so it will catch the Daylight Saving rules (and rule changes!) for you.

Finding ambiguous datetimes

At the end of lesson 2, we saw something anomalous in our bike trip duration data. Let’s see if we can identify what the problem might be.

The data is loaded as onebike_datetimes, and tz has already been imported from dateutil.

Instructions

• Loop over the trips in onebike_datetimes:
  • Print any rides whose start is ambiguous.
  • Print any rides whose end is ambiguous.

# Loop over trips
for trip in onebike_datetimes:
  # Rides with ambiguous start
  if tz.datetime_ambiguous(trip['start']):
    print("Ambiguous start at " + str(trip['start']))
  # Rides with ambiguous end
  if tz.datetime_ambiguous(trip['end']):
    print("Ambiguous end at " + str(trip['end']))

Ambiguous start at 2017-11-05 01:56:50-04:00
Ambiguous end at 2017-11-05 01:01:04-04:00

Avoid ambiguous datetimes in practice by storing datetimes in UTC.

Cleaning daylight saving data with fold

As we’ve just discovered, there is a ride in our data set which is being messed up by a Daylight Savings shift. Let’s clean up the data set so we actually have a correct minimum ride length. We can use the fact that we know the end of the ride happened after the beginning to fix up the duration messed up by the shift out of Daylight Savings.

Since Python does not handle tz.enfold() when doing arithmetic, we must put our datetime objects into UTC, where ambiguities have been resolved. 

onebike_datetimes is already loaded and in the right timezone. tzand timezone have been imported. Use tz.UTC for the timezone.

Instructions

• Complete the if statement to be true only when a ride’s startcomes after its end.
• When start is after end, call tz.enfold() on the end so you know it refers to the one after the daylight savings time change.
• After the if statement, convert the start and end to UTC so you can make a proper comparison.

trip_durations = []
for trip in onebike_datetimes:
  # When the start is later than the end, set the fold to be 1
  if trip['start'] > trip['end']:
    trip['end'] = tz.enfold(trip['end'])
  # Convert to UTC
  start = trip['start'].astimezone(tz.UTC)
  end = trip['end'].astimezone(tz.UTC)

  # Subtract the difference
  trip_length_seconds = (end-start).total_seconds()
  trip_durations.append(trip_length_seconds)

# Take the shortest trip duration
print("Shortest trip: " + str(min(trip_durations)))

Shortest trip: 116.0

Now you know how to handle some pretty gnarly edge cases in datetime data. To give a sense for how tricky these things are: we actually still don’t know how long the rides are which only started or ended in our ambiguous hour but not both. If you’re collecting data, store it in UTC or with a fixed UTC offset!

Loading a csv file in Pandas

The capital_onebike.csv file covers the October, November and December rides of the Capital Bikeshare bike W20529.

Here are the first two columns:

Instructions

• Import Pandas.
• Complete the call to read_csv() so that it correctly parses the date columns Start date and End date.

# Import pandas
import pandas as pd

# Load CSV into the rides variable
rides = pd.read_csv('capital-onebike.csv', 
                    parse_dates = ['Start date', 'End date'])

# Print the initial (0th) row
print(rides.iloc[0])

 Start date                        2017-10-01 15:23:25
    End date                          2017-10-01 15:26:26
    Start station number                            31038
    Start station                    Glebe Rd & 11th St N
    End station number                              31036
    End station             George Mason Dr & Wilson Blvd
    Bike number                                    W20529
    Member type                                    Member
    Name: 0, dtype: object

Did you know that pandas has a pd.read_excel(), pd.read_json(), and even a pd.read_clipboard() function to read tabular data that you’ve copied from a document or website? Most have date parsing functionality too.

Making timedelta columns

Earlier in this course, you wrote a loop to subtract datetime objects and determine how long our sample bike had been out of the docks. Now you’ll do the same thing with Pandas.

rides has already been loaded for you.

Instructions

• Subtract the Start date column from the End date column to get a Series of timedeltas; assign the result to ride_durations.
• Convert ride_durations into seconds and assign the result to the 'Duration' column of rides.

# Subtract the start date from the end date
ride_durations = rides['End date'] - rides['Start date']

# Convert the results to seconds
rides['Duration'] = ride_durations.dt.total_seconds()

print(rides['Duration'].head())

0     181.0
1    7622.0
2     343.0
3    1278.0
4    1277.0
Name: Duration, dtype: float64

Because Pandas supports method chaining, you could also perform this operation in one line: rides['Duration'] = (rides['End date'] - rides['Start date']).dt.total_seconds()

Summarizing datetime data in Pandas

ow many joyrides?

Suppose you have a theory that some people take long bike rides before putting their bike back in the same dock. Let’s call these rides “joyrides”.

You only have data on one bike, so while you can’t draw any bigger conclusions, it’s certainly worth a look.

Are there many joyrides? How long were they in our data set? Use the median instead of the mean, because we know there are some very long trips in our data set that might skew the answer, and the median is less sensitive to outliers.

Instructions

• Create a Pandas Series which is True when Start station and End station are the same, and assign the result to joyrides.
• Calculate the median duration of all rides.
• Calculate the median duration of joyrides.

# Create joyrides
joyrides = (rides['Start station'] == rides['End station'])

# Total number of joyrides
print("{} rides were joyrides".format(joyrides.sum()))

# Median of all rides
print("The median duration overall was {:.2f} seconds"\
      .format(rides['Duration'].median()))

# Median of joyrides
print("The median duration for joyrides was {:.2f} seconds"\
      .format(rides[joyrides]['Duration'].median()))

6 rides were joyrides
The median duration overall was 660.00 seconds
The median duration for joyrides was 2642.50 seconds

Pandas makes analyses like these concise to write and reason about. Writing this as a for loop would have been more complex.

It’s getting cold outside, W20529

Washington, D.C. has mild weather overall, but the average high temperature in October (68ºF / 20ºC) is certainly higher than the average high temperature in December (47ºF / 8ºC). People also travel more in December, and they work fewer days so they commute less.

How might the weather or the season have affected the length of bike trips?

Instructions 1/2

• 1
  • Resample rides to the daily level, based on the Start date column.
  • Plot the .size() of each result.

import matplotlib.pyplot as plt

# Resample rides to daily, take the size, plot the results
rides.resample('D', on = 'Start date')\
  .size()\
  .plot(ylim = [0, 15])

# Show the results
plt.show()

Since the daily time series is so noisy for this one bike, change the resampling to be monthly.

import matplotlib.pyplot as plt

# Resample rides to monthly, take the size, plot the results
rides.resample('M', on = 'Start date')\
  .size()\
  .plot(ylim = [0, 150])

# Show the results
plt.show()

As you can see, the pattern is clearer at the monthly level: there were fewer rides in November, and then fewer still in December, possibly because the temperature got colder.

se

Members vs casual riders over time

Riders can either be “Members”, meaning they pay yearly for the ability to take a bike at any time, or “Casual”, meaning they pay at the kiosk attached to the bike dock.

Do members and casual riders drop off at the same rate over October to December, or does one drop off faster than the other?

As before, rides has been loaded for you. You’re going to use the Pandas method .value_counts(), which returns the number of instances of each value in a Series. In this case, the counts of “Member” or “Casual”.

Instructions

• Set monthly_rides to be a resampled version of rides, by month, based on start date.
• Use the method .value_counts() to find out how many Member and Casual rides there were, and divide them by the total number of rides per month.

monthly_rides = rides.resample('M', on ='Start date')['Member type']

# Take the ratio of the .value_counts() over the total number of rides
print(monthly_rides.value_counts() / monthly_rides.size())

Start date  Member type
2017-10-31  Member         0.768519
            Casual         0.231481
2017-11-30  Member         0.825243
            Casual         0.174757
2017-12-31  Member         0.860759
            Casual         0.139241
Name: Member type, dtype: float64

Note that by default, .resample() labels Monthly resampling with the last day in the month and not the first. It certainly looks like the fraction of Casual riders went down as the number of rides dropped. With a little more digging, you could figure out if keeping Member rides only would be enough to stabilize the usage numbers throughout the fall.

Combining groupby() and resample()

A very powerful method in Pandas is .groupby(). Whereas .resample() groups rows by some time or date information, .groupby() groups rows based on the values in one or more columns. For example, rides.groupby('Member type').size() would tell us how many rides there were by member type in our entire DataFrame.

.resample() can be called after .groupby(). For example, how long was the median ride by month, and by Membership type?

Instructions

• Complete the .groupby() call to group by ‘Member type’, and the .resample() call to resample according to ‘Start date’, by month.
• Print the median Duration for each group.

# Group rides by member type, and resample to the month
grouped = rides.groupby('Member type')\
  .resample('M', on = 'Start date')

# Print the median duration for each group
print(grouped['Duration'].median())

<script.py> output:
    Member type  Start date
    Casual       2017-10-31    1636.0
                 2017-11-30    1159.5
                 2017-12-31     850.0
    Member       2017-10-31     671.0
                 2017-11-30     655.0
                 2017-12-31     387.5
    Name: Duration, dtype: float64

It looks like casual riders consistently took longer rides, but that both groups took shorter rides as the months went by. Note that, by combining grouping and resampling, you can answer a lot of questions about nearly any data set that includes time as a feature. Keep in mind that you can also group by more than one column at once.

Timezones in Pandas

Earlier in this course, you assigned a timezone to each datetime in a list. Now with Pandas you can do that with a single method call.

(Note that, just as before, your data set actually includes some ambiguous datetimes on account of daylight saving; for now, we’ll tell Pandas to not even try on those ones. Figuring them out would require more work.)

Instructions 1/2

• 1

Make the Start date column timezone aware by localizing it to 'America/New_York' while ignoring any ambiguous datetimes.

rides['Start date'] = rides['Start date'].dt.tz_localize('America/New_York', ambiguous='NaT')

# Print first value
print(rides['Start date'].iloc[0])

2017-10-01 15:23:25-04:00

• 2

Now switch the Start date column to the timezone 'Europe/London' using the .dt.tz_convert() method.

# Localize the Start date column to America/New_York
rides['Start date'] = rides['Start date'].dt.tz_localize('America/New_York', 
                                						 ambiguous='NaT')

# Print first value
print(rides['Start date'].iloc[0])

# Convert the Start date column to Europe/London
rides['Start date'] = rides['Start date'].dt.tz_convert('Europe/London')

# Print the new value
print(rides['Start date'].iloc[0])

2017-10-01 15:23:25-04:00
2017-10-01 20:23:25+01:00

dt.tz_convert() converts to a new timezone, whereas dt.tz_localize() sets a timezone in the first place. You now know how to deal with datetimes in Pandas.

How long per weekday?

Pandas has a number of datetime-related attributes within the .dt accessor. Many of them are ones you’ve encountered before, like .dt.month. Others are convenient and save time compared to standard Python, like .dt.weekday_name.

Instructions

• Add a new column to rides called 'Ride start weekday', which is the weekday of the Start date.
• Print the median ride duration for each weekday.

rides['Ride start weekday'] = rides['Start date'].dt.weekday_name

# Print the median trip time per weekday
print(rides.groupby('Ride start weekday')['Duration'].median())

Ride start weekday
Friday       724.5
Monday       810.5
Saturday     462.0
Sunday       902.5
Thursday     652.0
Tuesday      641.5
Wednesday    585.0
Name: Duration, dtype: float64

There are .dt attributes for all of the common things you might want to pull out of a datetime, such as the day, month, year, hour, and so on, and also some additional convenience ones, such as quarter and week of the year out of 52.

How long between rides?

For your final exercise, let’s take advantage of Pandas indexing to do something interesting. How much time elapsed between rides?

Instructions

• Calculate the difference in the Start date of the current row and the End date of the previous row and assign it to rides['Time since']. 
• Convert rides['Time since'] to seconds to make it easier to work with.
• Resample rides to be in monthly buckets according to the Start date.
• Divide the average by (60*60) to get the number of hours on average that W20529 waited in the dock before being picked up again.

rides['Time since'] = rides['Start date'] - (rides['End date'].shift(1))

# Move from a timedelta to a number of seconds, which is easier to work with
rides['Time since'] = rides['Time since'].dt.total_seconds()

# Resample to the month
monthly = rides.resample('M', on = 'Start date')

# Print the average hours between rides each month
print(monthly['Time since'].mean()/(60*60))

<script.py> output:
    Start date
    2017-10-31    5.519242
    2017-11-30    7.256443
    2017-12-31    9.202380
    Freq: M, Name: Time since, dtype: float64

As you can see, there are a huge number of Pandas tricks that let you express complex logic in just a few lines, assuming you understand how the indexes actually work. If you haven’t taken it yet, have you considered taking Pandas Foundations? In addition to lots of other useful Pandas information, it covers working with time series (like stock prices) in Pandas. Time series have many overlapping techniques with datetime data.