Convert HTML Table to an Array in Python

It's been quite a long time since my last update.

Currently, I'm working as an intern in one of the GPS. Well, life now is quite different from what was in school.

My first task is to extract some information from hundreds of HTML(4.0) files and fill the useful ones to Excel files.I heard about a famousGoogle's engineering decision:"Python where we can, C++ where we must".So I decided to learn python and adopt it  in this task.

Python 2.7.3 + beautifulSoup 4.1.3

An important procedure is to convert HTML tables to arrays like those in Matlab and OpenCV.

Here's what we've already got.

http://stackoverflow.com/questions/2870667/how-to-convert-an-html-table-to-an-array-in-python

def makelist(table):
  result = []
  allrows = table.findAll('tr')
  for row in allrows:
    result.append([])
    allcols = row.findAll('td')
    for col in allcols:
      thestrings = [unicode(s) for s in col.findAll(text=True)]
      thetext = ''.join(thestrings)
      result[-1].append(thetext)
  return result

For example

<TABLE ID = "3" BORDER=1 WIDTH="100%" COLS=3>
	<CAPTION ALIGN="LEFT">2.2 Prepare</CAPTION>
	<COLGROUP ALIGN="CENTER" SPAN=3>
	<TBODY>
		<TR>
			<TD COLSPAN = "3"><B>Plane Table</B></TD>
		</TR>
		<TR>
			<TH COLSPAN = "3">Prefiltration</TH>
		</TR>
		<TR>
			<TH>blabla<BR>[mm]</TH>
			<TH>blabla<BR>[mm]</TH>
			<TH>blabla<BR>[mm]</TH>
		</TR>
		<TR>
			<TD>2.50</TD>
			<TD>1.00</TD>
			<TD>3.50</TD>
		</TR>
</TABLE>

Each col in result would be 1, 0, 0, 3 using the above python code.However, one might wanna get result like 3, 0, 0, 3(then you can get 0, 3, 3, 0 just replace 'td' with 'th').

The key point to solve this problem is to get the attribute 'colspan'.

Here is the solution:

def makelist(table):
    result = []
    allrows = table.find_all('tr')
    for row in allrows:
        result.append([])
        allcols = row.find_all('td')
        for col in allcols:
            thestrings = [unicode(s) for s in col.find_all(text=True)]
            thetext = ''.join(thestrings)
            result[-1].append(thetext)
            if col.has_attr('colspan'):#if colspan != 0, filling blanks
                colspan = col.attrs['colspan']# or colspan = col['colspan']
                for i in range(int(colspan)-1):
                    result[-1].append(unicode(''))
    return result,len(allrows),int(table['cols'])

If  labels <td> and <th> are mixed in the same row, you can simply change

allcols = row.find_all('td')

to:

allcols = row.find_all(["td","th"])

then a table actually becomes 'one' table.