I am working on a website called Pedagogle.com which is intended to be a file sharing website for educators. The plan is, upload and categorize resources, and if enough people do this, and we have enough resources, then we end up with a website where teachers can go to find resources easily. The problem is that it takes a long time to upload and categorize each file, which leads to frustration from users. So far hardly any files have been uploaded, mostly because I think the process is a bit too difficult for the average teachers, and is definitely too time-consuming.
So I'm developing a module which will cut down the time it takes to share resources on my site. The basic work-flow I envisioned is, a teacher creates a ZIP archive for a folder of digital resources they have created, and uploads the ZIP. They then are presented with a form where they categorize all of the resources and save them to the database, with the option of deleting any files from our server right then that were included in the ZIP archive by error.
The first step in writing this module was to decide on how this was going to work conceptually for the user. It occurred to me that some users might not want to complete the categorization all at the same time, so that the files should be unarchived and stored in a temporary folder on the computer while the user is in the middle of the categorization, and that the process should be one that the user can come back to later if they like. This meant that the URL for the uploading of the ZIP should be different than the URL for managing the ZIP files to make the UI a bit easier to manage.
I implemented hook_perm() and hook_menu() in my module first like so:
<?php
/**
* Implementation of hook_perm().
*/
function zip_upload_perm() {
return array('upload zip files');
}
?>
<?php
/**
* Implementation of hook_menu().
*/
function zip_upload_menu() {
$items = array();
$items['zip_upload'] = array(
'title' => 'Upload ZIP',
'description' => 'Allows you to upload a zip of your resources and then categorize and save the resources',
'page callback' => 'drupal_get_form',
'page arguments' => array('zip_upload'),
'access callback' => 'user_access',
'access arguments' => array('upload zip files'),
);
$items['zip_upload/my'] = array(
'title' => 'My ZIP Uploads',
'description' => 'Categorize your uploaded files here and save them permanently',
'page callback' => 'drupal_get_form',
'page arguments' => array('zip_upload_recent'),
'access callback' => 'user_access',
'access arguments' => array('upload zip files'),
);
$items['admin/settings/zip_upload'] = array(
'title' => 'Zip Uploads',
'description' => 'Configure zip uploads for your site.',
'page callback' => 'drupal_get_form',
'page arguments' => array('zip_upload_settings_form'),
'access callback' => 'user_access',
'access arguments' => array('administer site configuration'),
'type' => MENU_NORMAL_ITEM,
);
return $items;
}
?>
I decided that even though my module was for a specific website, I should build it with some generalization in mind, just to make it easier to use the module for a different site. This is why I have a menu entry in my hook_menu() for a settings page.
Once I had my menu entries sorted out, the next step was to build both the forms for each path, but also the submit functions for these entries, which are the most difficult part of this module. The form for the settings page was extremely easy, especially as I could let Drupal handle the submission of the form by running it through system_settings_form.
<?php
/**
* Custom settings form for zip uploads
*/
function zip_upload_settings_form() {
$form['zip_upload_import_path'] = array(
'#type' => 'textfield',
'#title' => t('Zip Uploads path'),
'#description' => t('Choose to which directory inside your default files directory your zip uploads will be saved temporarily. Note that once the user finishes categorizing their resources, the zip files and the extracted files will be deleted automatically.'),
'#default_value' => variable_get('zip_upload_import_path', 'zip_upload'),
);
$form['zip_upload_allowed_file_types'] = array(
'#type' => 'textfield',
'#title' => t('Allowed extensions'),
'#description' => t('Choose which extensions of files are allowed to be contained within the zip archives. Separate each extension with a space.'),
'#default_value' => variable_get('zip_upload_allowed_file_types', 'jpeg jpg png gif'),
);
$types = node_get_types();
foreach ($types as $name => $type) {
$options[$name] = $name;
}
$form['zip_upload_node_type'] = array(
'#type' => 'select',
'#title' => t('Choose node type'),
'#description' => t('You must associate each file uploaded with a node type that has a either an upload field or a filefield attached to it.'),
'#options' => $options,
'#default_value' => variable_get('zip_upload_node_type', 'page'),
);
return system_settings_form($form);
}
?>
The upload ZIP form was also very easy, I just had to remember to make the form have the right information initially, so hence $form['#attributes'] = array("enctype" => "multipart/form-data");.
<?php
/**
* Zip Upload form
*/
function zip_upload() {
$form = array();
$form['#attributes'] = array("enctype" => "multipart/form-data");
$form['zipfile'] = array(
'#type' => 'file',
'#title' => t('Zip'),
'#size' => 40,
'#description' => t('Click "Browse..." to select a Zip file to upload.'),
'#weight' => -3,
);
$form['hash_directory'] = array(
'#type' => 'value',
'#value' => md5(mktime()),
);
$form['file'] = array(
'#type' => 'value',
'#value' => '',
);
$form['buttons']['submit'] = array(
'#type' => 'submit',
'#value' => t('Import zip'),
);
return $form;
}
?>
The hard part about the ZIP upload form was the validating function and the submit function. I decided the best way of handling this section would be to extract the ZIP during validation (after verifying it was a legitimate ZIP archive) into a temporary folder and then go through each file and check the extension, and remove all files with improper file extensions immediately. In the submit handler, I save the information about the directory for later, so that the user can come back to the next step, and then redirect the user to the next form.
<?php
/**
* Validate upload form
*
* @function
* Confirm the user has uploaded a zipped file first
* then we extract the file to a temporary folder and remove any undesired files.
*/
function zip_upload_validate($form, &$form_state) {
$dirpath = file_create_path(variable_get('zip_upload_import_path', 'zip_upload'));
if (file_check_directory($dirpath)) {
if ($file = file_save_upload('zipfile', array('file_validate_extensions' => array('zip')),$dirpath . $file->filename)) {
if (!$file) {
form_set_error('zipfile', t('Zip file not uploaded'));
}
else {
// try to avoid php's script timeout with large files or a slow machine
if (!ini_get('safe_mode')) {
set_time_limit(0);
}
// extract the zip file into the subdirectory
$zip = new ZipArchive;
if ($zip->open($dirpath . '/' . $file->filename) === TRUE) {
$directory = file_create_path($dirpath . '/' . $form_state['values']['hash_directory']);
$zip->extractTo($directory);
$zip->close();
file_delete($dirpath .'/'. $file->filename);
// check here for bad files and remove them?
$allowed_extensions = drupal_strtolower(variable_get('zip_upload_allowed_file_types', 'jpeg jpg png gif'));
$files = file_scan_directory($directory, '.*');
// This is crucial. We need to remove disallowed files.
$invalid = 0;
foreach ($files as $filename => $file) {
$errors = file_validate_extensions($files[$filename], $allowed_extensions);
if ($errors) {
$name = array_pop(explode("/", $filename));
// warn the user
drupal_set_message("Skipping " . check_plain($name) ." because extension isn't allowed", 'error');
// remove the file
file_delete($filename);
unset($files[$filename]);
$invalid++;
}
}
if ($invalid) {
drupal_set_message("There were $invalid errors while extracting your archive.", 'error');
}
elseif (!empty($files)) {
drupal_set_message(t('Zip file uploaded and extracted.'), 'success');
}
if (empty($files)) {
// remove empty subdirectories
_zip_upload_rmdir_recurse($directory);
// remove the now empty top directory
rmdir($directory);
form_set_error('zipfile', t('Your archive has no files with valid extensions in it. Please upload a different archive and try again.'));
}
}
else {
drupal_set_message(t('There was a problem trying to extract the zip archive to the temporary folder. Report this issue to your system administrator.'), 'error');
}
}
}
}
}
/**
* Submit upload form
*
* @function
* Here we store the information about the temporary folder and send the user to a
* form to categorize and give a description to each file.
*/
function zip_upload_submit($form, &$form_state) {
// save the zip file location and current user to the database for the next step
global $user;
db_query("INSERT INTO {zip_upload} (uid, hash_directory) VALUES (%d, '%s')", $user->uid, $form_state['values']['hash_directory']);
return drupal_goto('zip_upload/my');
}
?>
Once the files are sorted onto the server and stored in a location which is difficult for the user to access, we then want the user to decide on a categorization for each file and give each file a quick description. Since I've stored the names of the temporary directories in my database at the previous step on a per user basis, this allows me to keep each set of uploaded files separate, and allows users to come back to the final form whenever they want. I decided on a gigantic table for my form with the vocabularies as select elements, the description for each file as a textfield, and then a checkbox next to each file name in each table so that the user could choose which files to either save permanently or delete.
<?php
/**
* Categorize recent resources form
*/
function zip_upload_recent() {
$form = array();
$vocabularies = _zip_upload_get_vocabularies();
// get the current user
global $user;
// look up any folders the user has created because of a zip upload
$result = db_query("SELECT hash_directory FROM {zip_upload} WHERE uid = %d", $user->uid);
$form['files'] = array(
'#type' => 'markup',
'#title' => t('Uploaded files'),
'#description' => t('Categorize each file, then click on Save'),
'#tree' => TRUE,
);
// at this stage files with disallowed extensions should be removed already.
while ($row = db_fetch_array($result)) {
$dirpath = file_directory_path() .'/'. variable_get('zip_upload_import_path', 'zip_upload') .'/'. $row['hash_directory'];
if (file_check_directory($dirpath)) {
$directory = $dirpath;
};
$files = file_scan_directory($directory, '.*');
// add an index to each textfield
$index = 0;
foreach ($files as $filename => $file) {
$form['files'][$index]['checked'] = array(
'#type' => 'checkbox',
'#default_value' => 0,
'#attributes' => array('class' => 'checked'),
);
$form['files'][$index]['filename'] = array(
'#type' => 'value',
'#value' => $filename,
);
// add a description
$form['files'][$index]['description'] = array(
'#type' => 'textfield',
'#size' => 20,
'#maxlength' => 255,
'#value' => '',
'#attributes' => array('class' => 'textfield-' . $index),
);
// now we include the taxonomy terms, bulk of this copied from taxonomy.module
foreach ($vocabularies as $vid => $name) {
// add a taxonomy select
$form['files'][$index]['taxonomy'][$vid] = taxonomy_form($vid, 0, ' ');
// add a class to each taxonomy select to make it easier to grab with jQuery
$form['files'][$index]['taxonomy'][$vid]['#attributes'] = array('class' => 'taxonomy-' . $vid);
// remove the title from each field
unset($form['files'][$index]['taxonomy'][$vid]['#title']);
}
$index++;
}
// remove this directory to help keep the folder clean.
if ($index === 0) {
if (file_check_directory($directory)) {
//_zip_upload_rmdir_recurse($directory);
rmdir($directory);
}
db_query("DELETE FROM {zip_upload} WHERE hash_directory = '%s'", $row['hash_directory']);
}
}
$form['number_of_files'] = array(
'#type' => 'value',
'#value' => $index,
);
$form['submit'] = array(
'#type' => 'submit',
'#value' => t('Save'),
);
$form['delete'] = array(
'#type' => 'submit',
'#value' => t('Delete'),
);
return $form;
}
?>
Theming this form was relatively straight forward, I just converted by fields into individual rows and sent off most of the form to be themed as a table. At this stage I also added a CSS file and a JS file for later customization which I suspected was going to be necessary.
<?php
/**
* Theme the upload categorization form
*/
function theme_zip_upload_recent($form) {
$path = drupal_get_path('module', 'zip_upload');
drupal_add_js($path . '/js/zip_upload.js');
drupal_add_css($path . '/css/zip_upload.css');
$vocabularies = _zip_upload_get_vocabularies();
// build the header of the table
$header = array(
t('Select'),
t('File name')
);
foreach ($vocabularies as $vid => $vocabulary) {
$header[] = check_plain($vocabulary);
}
// unlikely we'll have more than 1000 vocubularies created on the same site
$header[] = t('Description');
$rows = array();
for ($index = 0; $index < $form['number_of_files']['#value']; $index++) {
$row = array(
'checked' => drupal_render($form['files'][$index]['checked']),
);
$name = array_pop(explode("/", $form['files'][$index]['filename']['#value']));
// already filtered
$row['filename'] = check_plain($name);
foreach ($vocabularies as $vid => $vocabulary) {
$row[$vid] = drupal_render($form['files'][$index]['taxonomy'][$vid]);
}
$row['description'] = drupal_render($form['files'][$index]['description']);
$rows[] = $row;
}
if (empty($rows)) {
return t('You do not have any files left to perform operations on.');
}
$output .= theme('table', $header, $rows, array('class' => 'zip-upload-table'));
$output .= drupal_render($form);
return $output;
}
?>
The next step was to build the submit function for the form. A validation function was unnecessary in this case since I had already validated the file extensions, and my site fixes the input filter for the description at the default filter. Also, the user doesn't actually have access to the file locations, so I wasn't worried about the files being switched somehow at this stage. My submit function basically splits into two cases, one where the user wants to save the file permanently, and one where the user wants to delete the file. In both cases, the final step is to check if the file is in an now empty sub-directory and delete that directory. One issue I am still having is creating a function to check if the entire temporary directory is empty, and removing it if it is.
I decided to use node_submit and node_save here to save the node information rather than drupal_execute because I have found the first two functions easier to work with, and they serve my purpose well. Finally, I was unsure if the validation on the file field would work properly given that the uploaded file being attached to the node wasn't in the official temporary directory. In any case all of the right hooks seem to fire using this code, and my users get points (using the User Points module) as expected.
<?php
/**
* Submit function
*/
function zip_upload_recent_submit($form, &$form_state) {
// aliased for readability
$form_values = $form_state['values'];
if ($form_values['op'] == 'Save') {
global $user;
// figure out what the proper destination is for this file
if (module_exists('token')) {
$types = array(
'user' => $user,
);
if (module_exists('date')) {
$types['field'] = array(0 => array(
'value2' => 'now',
));
}
// do I need to hard-code this? I suspect not...
$dest = token_replace_multiple('[yyyy]/[mm]/[dd]/[uid]', $types);
}
foreach ($form_values['files'] as $index => $file) {
$filename = $file['filename'];
if ($file['checked']) {
// get the name of the file
$name = array_pop(explode("/", $filename));
// create a title for the node
$title = str_replace("-", " ", check_plain(array_shift(explode(".", $name))));
$destination = file_directory_path() .'/'. $dest .'/'. $name;
$path = file_directory_path() .'/'. $dest;
// get some information about the file
$filemime = file_get_mimetype($filename);
$filesize = filesize($filename);
// create the destination directory, if it does not already exist
if (file_check_directory($path) || mkdir($path, 0775, TRUE)) {
// TO DO: Fix this so that empty directories are removed. Maybe a call to a single function
// at the end instead?
// check to see if the directory is now empty
$source = rtrim(str_replace($name, "", $filename), "/");
// move the file
if(file_move($filename, $destination)) {
$files = file_scan_directory($source, ".*");
// if it is empty, remove the directory
if (empty($files)) {
rmdir($source);
}
// the query will be the same for each file, just the placeholders will be different
$query = "INSERT INTO {files} (uid, filename, filepath, filemime, filesize, status, timestamp) VALUES (%d, '%s', '%s', '%s', '%s', 1, %d)";
// insert an entry into the files table
db_query($query, $user->uid, $name, $destination, $filemime, $filesize, time());
// get the fid, since we know the full filepath will be unique
$fid = db_result(db_query("SELECT fid FROM {files} WHERE filepath = '%s'", $destination));
// build the node here
$node = array(
'title' => $title,
'type' => variable_get('zip_upload_node_type', 'page'),
'uid' => $user->uid,
'body' => $file['description'],
'promote' => 0,
'status' => 1,
'pathauto_perform_alias' => 1,
'field_upload' => array(
0 => array(
'fid' => $fid,
'list' => 1,
'data' => 0,
'filename' => $name,
'filepath' => $destination,
'filemime' => $filemime,
'source' => 'field_upload_0',
'destination' => $destination,
'taxonomy' => $file['taxonomy'],
'filesize' => $filesize,
'uid' => $user->uid,
'status' => 1,
),
),
);
if ($node = node_submit($node)) {
$node->uid = $user->uid;
$node->taxonomy = $file['taxonomy'];
node_save($node);
drupal_set_message(check_plain($title) . " has been saved.", "success");
}
else {
drupal_set_message("There was an error saving " . check_plain($title) . ".");
}
}
else {
drupal_set_message(t('There was a problem moving your file. Talk to your system administrators about fixing this problem.'), 'error');
}
}
else {
drupal_set_message(t('There was a problem moving your file. Talk to your system administrators about fixing this problem.'), 'error');
}
}
}
}
elseif ($form_values['op'] == 'Delete') {
foreach ($form_values['files'] as $filename => $file) {
if ($file['checked']) {
// delete the file in question
file_delete($filename);
// check to see if the directory is now empty
$fileArray = explode("/", $filename);
array_pop($fileArray);
$directory = implode($fileArray, "/");
$files = file_scan_directory($directory, ".*");
// if it is empty, remove the directory
if (empty($files)) {
rmdir($directory);
}
}
}
}
return;
}
?>
There is one additional helper function I use to return the vocabularies for a given node type, which I suspect already exists in the Taxonomy module, but I didn't find it. This helper function was just a bit easier to use and customized for my module.
<?php
/**
* Internal function to return the categories for our fixed node type.
*/
function _zip_upload_get_vocabularies() {
$type = variable_get('zip_upload_node_type', 'page');
$types = node_get_types();
foreach ($types as $name => $value) {
$options[$name] = $name;
}
// get the vocabularies associated with this node type
$c = db_query(db_rewrite_sql("SELECT v.* FROM {vocabulary} v INNER JOIN {vocabulary_node_types} n ON v.vid = n.vid WHERE n.type = '%s' ORDER BY v.weight, v.name", 'v', 'vid'), $options[$type]);
while ($vocabulary = db_fetch_object($c)) {
$vocabularies[$vocabulary->vid] = $vocabulary->name;
$form['vocabularies'][$vocabulary->vid] = array(
'#type' => 'value',
'#value' => check_plain($vocabulary->name),
);
}
return $vocabularies;
}
?>
I was having problems with the theme function being recognized, until I realized that contrary to the instructions in the FAPI documentation on http://api.drupal.org , theme functions are not automatically discovered for functions which define Drupal forms, so I had to implement hook_theme as well.
<?php
/**
* Implementation of hook_theme().
*/
function zip_upload_theme($existing, $type, $theme, $path) {
return array(
'zip_upload_recent' => array(
'arguments' => array(),
),
);
}
?>
Once I had my form working, I decided I needed to add some JavaScript to make it a bit easier to use. Categorizing 100 files, all of which are in the same categories could be a bit tedious, so I added some JS to make it easier to apply the same categorization of Topic, Type and Age level (the three vocabularies used to categorize resources on Pedagogle. The JS was relatively easy to build but I had to refer to the documentation over at JQuery a few times in order to get it right.
I also found myself going back to the PHP code a few times in order to add class and id attributes where appropriate to make it easier to select elements to manipulate using jQuery.
// $Id$
// check to make sure JS is enabled and functional
if (Drupal.jsEnabled) {
// add our stuff inside the header of the table
$(document).ready(function() {
// first add a checkbox to the top of the first column
$("table.zip-upload-table th:first").each(function() {
// set a width on the first column
$(this).css({'width' : '100px'});
// add a checkbox
$(this).prepend('<input type="checkbox" id="check-all" /> ');
// make the checkbox select and deselect all other checkboxes in this table
$("#check-all").click(function() {
$("input.checked").attr("checked", $(this).attr("checked"));
});
});
// now add a taxonomy select to the top of the 3rd, 4th, and 5th columns
for (var i = 2; i <= 4; i++) {
$("table.zip-upload-table th:eq(" + i + ")").each(function() {
var text = $(this).text();
var tableHeader = this;
$(this).html($("table.zip-upload-table td:eq(" + i + ")").html());
$(this).find("option:eq(0)").css({"font-weight" : "bold"}).text("Choose " + text);
$(this).find("select:eq(0)").change(function() {
var index = 1*$(this).attr("class").replace("form-select taxonomy-", "") + 1;
$("table.zip-upload-table tr").find("td:eq(" + index + ") select:eq(0)").val($(this).val());
});
});
}
// now make the textfield's easier to use by adding up, down arrow and tab navigation
$("table.zip-upload-table input.form-text").focus(function () {
// add a keydown event on focus
$(this).keydown(function (event) {
// get the current index
var thisIndex = 1*$(this).attr("class").replace("form-text textfield-", "");
// get the total number of textfields
var size = $("table.zip-upload-table input.form-text").size();
// get the event which is browser dependent
var e = event.which || event.keyCode;
// which key has been hit
switch (e) {
// enter key
case 13:
// tab key
case 9:
// down arrow
case 40:
// increase the index by 1, then use modulo arithmetic to prevent potential over-run
thisIndex = (thisIndex + 1) % size;
// focus on the new textfield
$("table.zip-upload-table input.form-text").get(thisIndex).focus();
break;
// up arrow
case 38:
// decrease the index by 1, then make sure that negative values jump up to the other end
thisIndex = ((thisIndex - 1) < 0) ? thisIndex + size - 1: thisIndex - 1;
// focus on the new textfield
$("table.zip-upload-table input.form-text").get(thisIndex).focus();
break;
// nothing to do unless the event is receiving an up or down arrow
default:
return true;
}
// remove the keydown until the next time this textfield has focus
$(this).unbind('keydown');
// prevent bubbling
return false;
});
});
// just in case the user clicks between boxes, remove the keydown from before
$("table.zip-upload-table input.form-text").blur(function () {
$(this).unbind('keydown');
});
});
}
The total development time for this module was about a day, but it still has a couple of small things I need to fix. For example, I'd like to be able to clean up the empty temporary directories at some stage, either through hook_cron, or as soon as it is emptied. On a Mac, when you ZIP up a directory using most of the tools available (including the default archiving tool provided), it adds other directories not present in a Windows archive, like __MASOSX and _trashes, etc... So my code needs to ignore files in those directories, which I'll probably do using regular expressions, quietly during the validation stage, since these are hidden directories in the Mac OS X, and most users are blissfully unaware of their existence.
Anyway, if anyone can help me with the recursive checker for empty sub-directories, which also then deletes those sub-directories, that would be great. If there are any other OS specific problems I should be aware of in Zip archives, that would also be handy. For instance, I'm not sure what happens if a user sets permissions on the files in their archive in Linux, and then uploads the Zip. Do the permissions stick using my current code? I'll probably have to check before I let everyone have access to this form. For now, I've set up a "trusted user" role, and assigned the "upload zip archives" permission to this role, and allow individual users to access this form as I see fit.
Finally, the file doesn't include any embedded help text right now because I'm still working on it, and because I've got a module I created a while ago which allows me to edit help text for any path on the fly. Once I've got some good solid help text written, I'll include it in this module as well.
Hope you find this tutorial useful, and let me know where I can make some improvements in my code, as I am most definitely not a professional PHP programmer.
]]>
