2008-08-22

HowTo retrieve URL content using Python

With the following code you can fetch any content from web using http or ftp:

def fetchContent(self, url): try: fetchResult = urlfetch.fetch(url) except: return 'Exception: Not an URL' if fetchResult.status_code == 200: encTypeRaw = fetchResult.headers['content-type'] encStartIdx = encTypeRaw.find('=') encType = encTypeRaw[encStartIdx + 1:] fetchResultContent = fetchResult.content return fetchResultContent.decode(encType) return 'Warning: URL could not be retrieved successfully'