Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What?s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa?s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available ?warts and all? for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you?re hoping to do with it. We may not be able to say ?yes? to all requests, since we?re just figuring out whether this is a good idea, but everyone will be considered.
Chinua Achebe once fulminated: “'Art for art's sake is just another piece of deodorised dog s**t'.”
While Achebe went too far in my view - art for art's sake can be great fun; I'm in the movie business after all - it's lamentable that art in the West can too often be dismissed as a decorative frill or a recreational distraction from the doldrums of our comfortable Western existence.
But not in Africa, where ravaged by war, famine, disease and domestic violence, the arts are being invoked to help child victims of rape, Aids, and war to cope with and confront conditions that no human being was designed to endure.
Of course, as more UN aid convoys rumble into the rebel-held Democratic Republic of Congo, it's impossible to deny that what is needed most in crisis-stricken Africa is hardly film, music and theatre but water, food and medicine. But after a catastrophe, these essential staples of life are not enough to restore it fully. Through the arts, young children living in catastrophic circumstances need to be given something to live for and something through which to express their lives.
For the past three years, I've helped to nurture a new arts-led voluntary initiative called Dramatic Need, which works with children living in abysmal conditions in sub-Sarahan Africa. Last year the organisation encountered four-year-old Bongani, an orphan whose mother had died of Aids when he was 3. This little boy, along with 30 others, was being cared for in a crèche in a squatter camp in the North West Province of South Africa. With pitiful resources, their teacher and carer struggled to provide these kids with anything that engaged their attention. They were not learning, or behaving. The teacher described four-year-olds violently attacking each other and on one occasion, her. Because these kids were so young, volunteers focused on helping them to make coloured playdough and paint murals on the corrugated iron walls outside their school.
One day, they helped the children to make papier-mâché masks. “Make your mask as scary as possible” was the brief. At the end of the day, as the teacher began to collect 34 Cubist masterpieces - eyes askew, noses where the ears should be - Bongani, this small, typically placid four-year-old, went berserk, screaming, shouting, lashing out, repeating the same sentence in Tswana over and over again. His teacher translated: “He's saying ‘Don't take her away, don't take my mum away again'.”
Bongani had shaped his grief into a mask of newspaper, paint and wallpaper paste. I am neither clairvoyant nor therapist, but I was told about the effect it had on him when he was allowed to take it home. They said the kid practically flew.
For children - and for many adults - art plays a vital role in helping them to express feelings and difficulties that they aren't otherwise able to articulate. Its importance is never greater than in post-conflict conditions. Of course, water, food, and first aid are essential during a crisis, but none of these things can restore human dignity to a person dying from disease or help a rape victim to cope with their outrage.
To suggest that the only things that maintain our humanity are those that serve our biological needs seems to me palpably incorrect. We are not just what we eat. We are also what we feel, what we fear, what we love and what we hate. Unexpressed tensions find their strength in violence. I look at the Congo now; if there is not a means to move beyond the hatred of the past, we will never move past violence.
I am not suggesting that we drop paint-brushes on Goma. But I am suggesting that post-conflict relief should look to means of coping with and expressing individual trauma, and that the arts can play a vital role. Whether visual or performance-based, they can be psychotherapeutic. They allow people to participate in their own recovery, help them to relocate and resuscitate their sense of self. It's not for nothing that the arts are called the humanities; they humanise us.
Often the arts can help children to confront - as well as comprehend - deep-seated, community-held taboos, such as HIV and domestic violence, which, in some rural areas where Dramatic Need works, affect about 80 per cent of women. In May this year we ran a drama workshop with a group of teenagers in South Africa, encouraging them to pick a challenging issue that mattered to them. One of the girls chose domestic violence, or “anger at home” as she called it: what emerged later was this 14-year-old had chosen to use her play to dramatise her own rape, a secret she confided in a volunteer afterwards.
It was clear that she'd invested a huge part of herself emotionally, and despite breaking down several times during rehearsals, she was determined to carry on. After performing the play to her unsuspecting peers, she thanked the volunteer for giving her the chance to “make people know” the horrific things to which she'd been subjected. While she'd not revealed its autobiographical dimension, her drama had helped to exorcise a demon that had thrived on silence.
But the escape that artistic self-expression provides African children is not only imaginary or psychological. For the extremely talented, it provides a way of getting out of the ghetto, of fulfilling unimagined ambitions, and unlocking unknown creative potential.
Seboko Phillemon Morobi, an 18-year-old boy from Viljoenskroon, in South Africa, lost his right hand in an accident and his parents to HIV when he was very young. Despite being highly intelligent and having taught himself to speak English, his disability, in a dominant agricultural community with high unemployment, meant he had little chance of finding work and would be destined to live in abject poverty. When they found Seboko, he was sleeping in a neighbour's barn, and walking six miles each day to school.
He showed an extraordinary interest in a film-making workshop, using mobile phones to capture and cut mini-documentaries about things the children cared about. This young guy - who'd never seen, let alone edited a film before - displayed an instinctive genius for shaping narrative. Dramatic Need felt so strongly about this that the charity has just secured him an internship at Design Republic, Johannesburg's leading media agency. We are now fundraising to help him to get there.
For these African children, then, art is not an adjunct to life but can transform it. It has an educative, restorative, reorienting power, not despite but precisely because of their desperate circumstances.
I am currently publicising Slumdog Millionaire, a film about a child escaping the slums of Bombay. During the filming I saw plenty of what the world's poorest children endure and, frankly, anything that lightens their load is important. In these rural and township parts of southern Africa - where tragedy is the rule, not the exception - the essential, humanitarian relevance of the arts is unequivocal.
Danny Boyle is the director of Slumdog Millionaire. Dramatic Need launches in London on Thursday. For information, go to www.dramaticneed.org
Contact us | Terms and Conditions | Privacy Policy | Site Map | FAQ | Syndication | Advertising
© Times Newspapers Ltd 2010 Registered in England No. 894646 Registered office: 3 Thomas More Square, London, E98 1XY