Lost password?    Lost username?    Create an account
OverviewDownloadsForumSupport
CSV Improved

CSV Improved

  • Free upgrades for 1 year
  • Extended replacement rules
  • Advanced XML support
  • Multiple component support
  • Automatic import/export
  • Advanced import/export features
  • Forum access for 1 year
  • Forum access for 10 days
Welcome, Guest
Username: Password: Remember me
Looking for a useful answer?

Read this first and make sure you checked all steps: www.csvimproved.com/frequently-asked-questions/635-troubleshooting-import-errors

Don't just say it doesn't work but describe in detail what is happening and add the following information:
* VirtueMart version
* CSVI version (Latest is not a version number)
* Sample of the file being imported
* Collected debug information

Without this information you won't get a useful answer. If no debug log is attached, you will get a request to post it.

Where to get the debug information?

A tutorial (www.csvimproved.com/frequently-asked-questions/390-how-to-collect-debug-information) has been written to help you collect your debug information. For CSVI 3, there is a video available (www.csvimproved.com/downloads/file/18-collect-debug-information)

TOPIC: Automacally dowload and import dats

Automacally dowload and import dats 1 year 3 weeks ago #28042

  • vanaveno
  • vanaveno's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 38
  • Karma: 0
I would like to do of script for automatically download and import dats. I made some code modifications, I added code to download a file from the server and I would like to modify it so that I could use as a script for crontab.

I can not program in php. Could someone advise me how to modify the code csvi_cutter.php in order to use it to crontab without forms.

Thank you very much.

This is my changes
<?php
/**
 * CSVI Cutter
 *
 * Tool to cut a CSV file into multiple files
 *
 * @package CSVImproved
 * @subpackage Tools
 * @author Roland Dalmulder
 * @link http://www.csvimproved.com
 * @copyright Copyright (C) 2006 - 2009 RolandD Cyber Produksi
 * @version $Id: csvi_cutter.php 999 2009-09-17 08:01:20Z Suami $
 */
?>
<html>
<head>
	<title>CSV Improved Cutter</title>
	<script type="text/javascript">
	<!--
	function expand_layer(sid,show) {
		var sbox = document.getElementById(sid);
		var sbox_style = sbox.style;
		if (show===true) {
			sbox_style.display='block';
		}
		else if (show===false) {
			sbox_style.display='none';
		}
		else {
			if ((sbox_style.display=='none')||(sbox_style.display=='')) {
				sbox_style.display='block';
			}
			else {
				sbox_style.display='none';
			}
		}
		return false;
	}
	-->
	</script>
	<style type="text/css">
	<!--
	#errormsg {
		font-size: 20px;
		background: #FF6F78;
	}
	#result {
		font-size: 16px;
		background: #59D961;
	}
	#title, #logo{
		display: block;
		font-size: 30px;
		font-weight: bold;
		text-align: center;
	}
	label {
		width: 12em;
		float: left;
		text-align: right;
		margin-right: 0.5em;
		display: block
	}
	.submit {
		margin-left: 12.5em;
	}
	input {
		color: #0C609C;
		font-weight: bold;
		background: #FFCD8B;
		border: 1px solid #0C609C;
	}
 
	.submit input {
		color: #000000;
		background: #FFCD8B;
		border: 2px outset #d7b9c9;
	}
	li {
		padding: 5px;
	}
	#usage {
		display: none;
	}
	.subtitle {
		font-size: 12px;
	}
	a {
		text-decoration: none;
		color: #0C609C;
	}
	-->
	</style>
</head>
<body>
<?php
 
/* Download availibility */
$source_a = "http://www.my-download-link...";
 $destination_a = "Stock_CSV.zip";
 
 $data_a = file_get_contents($source_a);
 $file_a = fopen($destination_a, "w+");
 fputs($file_a, $data_a);
 fclose($file_a);
 chmod("Stock_CSV.zip", 0777);
 
 
 /* Unzip availibilty */
   $destination_a = ''; //blank for root folder
 
  $zip_a = new ZipArchive;
 
  $zip_file_a = $zip_a->open('Stock_CSV.zip');
 
  if ( $zip_file_a ) {
 
  $zip_a->extractTo(dirname(__FILE__).$destination_a);
 
  $zip_a->close();
 
  echo 'files successfully extracted to
      <strong>'.dirname(__FILE__).$destination_a.'
     </strong> directory.';
  }
  else {
  echo 'No donuts for you. :(';
  }
  chmod("Stock_CSV.csv", 0777);
 
 
 
 
?>
 
 
 
<?php
class CsvCut {
 
	/* Private variables */
	private $_filename = 'Stock.csv';
	private $_filecount = 1;
	private $_column_headers = null;
	private $_csv_data = null;
	private $_columnheader = null;
	private $_maxrows = 850;
	private $_fielddelimiter = '\t';
	private $_textenclosure = '"';
	private $_fielddelimiter_target = '\t';
	private $_textenclosure_target = '"';
	private $_outputfolder = '/var/www/stock/';
	private $_original_filename = 'Stock';
	private $_errormsg = array();
 
 
	/* Public variables */
 
	/**
	 * Initialize the class
	 */
	public function __construct() {
		/* Do we need to process the form data or show the form */
		$this->showForm();
		if (isset($_POST['process'])) {
			if (!$this->processForm()) {
				if (count($this->_errormsg) > 0) {
					?>
 
 
 
          <div id="errormsg"><?php
						foreach ($this->_errormsg as $message) {
							echo $message.'<br />';
						}
					?></div><?php
				}
			}
		}
	}
 
	/**
	 * Show a form with user options
	 */
	private function showForm() {
	 	?>
	 	<div id="logo"><img src="csvi_logo.png" alt="CSV Improved" /></div>
	 	<div id="title"><span class="subtitle">presents:</span><br />CSV Improved Cutter</div>
	 	<br />
	 	<form name="csvcut" method="post" enctype="multipart/form-data">
			<label for="filename">Filename: </label>
				<input type="file" id="filename" name="filename" size="80" /><br /><br />
			<label for="outputfolder">Output folder: </label>
				<input type="text" id="outputfolder" name="outputfolder" size="80" /><br /><br />
			<label for="columnheader">File has column headers: </label>
				<input type="checkbox" id="columnheader" name="columnheader" value="1" checked="checked" /><br /><br />
			<label for="maxrows">Maximum rows: </label>
				<input type="text" id="maxrows" name="maxrows" size="5" /><br /><br />
			<label for="fielddelimiter">Field delimiter: </label>
				<input type="text" id="fielddelimiter" name="fielddelimiter" size="5" /><br /><br />
			<label for="textenclosure">Text enclosure: </label>
				<input type="text" id="textenclosure" name="textenclosure" size="5" /><br /><br />
			<label for="fielddelimiter">Field delimiter target: </label>
				<input type="text" id="fielddelimiter_target" name="fielddelimiter_target" size="5" /><br /><br />
			<label for="textenclosure">Text enclosure target: </label>
				<input type="text" id="textenclosure_target" name="textenclosure_target" size="5" /><br /><br />
			<div class="submit"><input type="submit" value="Create files"> <input type="reset" value="Reset"></div>
		  <input type="hidden" id="process" name="process" value="" />
	 	</form>
	 	<a href="csv_cut.php" onclick="expand_layer('usage'); return false;">How to use?</a>
 
	 	<?php
	}
 
	/**
	 * Process a file
	 */
	private function processForm() {
		/* Assign the local values */
 
		/* Get the posted values */
	  if (!empty($_POST['filename'])) $this->_maxrows = $_POST['filename'];
		if (!empty($_POST['maxrows'])) $this->_maxrows = $_POST['maxrows'];
		if (!isset($_POST['columnheader'])) $this->_columnheader = false;
		if (!empty($_POST['fielddelimiter'])) $this->_fielddelimiter = $_POST['fielddelimiter'];
		if (!empty($_POST['textenclosure'])) $this->_textenclosure = $_POST['textenclosure'];
		if (!empty($_POST['fielddelimiter_target'])) $this->_fielddelimiter_target = $_POST['fielddelimiter_target'];
		else $this->_fielddelimiter_target = $this->_fielddelimiter;
		if (!empty($_POST['textenclosure_target'])) $this->_textenclosure_target = $_POST['textenclosure_target'];
		else $this->_textenclosure_target = $this->_textenclosure;
		if (!empty($_POST['outputfolder'])) $this->_outputfolder = $_POST['outputfolder'];
 
 
 
		/* Check if we can read the file */
		$row = 1;
		$handleread = fopen($this->_filename, "r");
		if ($handleread) {
			?>
			<div id="result">
			<?php
			while (($data = fgetcsv($handleread, 0, $this->_fielddelimiter, $this->_textenclosure)) !== FALSE) {
				$num = count($data);
				$row++;
				for ($c=0; $c < $num; $c++) {
					$this->_csv_data .= $this->_textenclosure_target.str_replace($this->_textenclosure_target, $this->_textenclosure_target.$this->_textenclosure_target, $data[$c]).$this->_textenclosure_target;
					if ($c < $num && $num - $c > 1) $this->_csv_data .= $this->_fielddelimiter_target;
					if ($num - $c == 1) $this->_csv_data .= chr(10);
 
					/* Get the column headers from the first file */
					if ($this->_columnheader && $row == 2 && $this->_filecount == 1 && $num - $c == 1) {
						$this->_column_headers = $this->_csv_data;
					}
				}
 
				if ($row == $this->_maxrows) {
					$result = $this->writeData();
					if ($result) {
						$this->_filecount++;
						$row = 0;
						$this->_csv_data = null;
					}
					else {
						echo 'Could not write file';
						exit;
					}
				}
			}
			if (!$data && strlen($this->_csv_data) > 0) {
				$this->writeData();
				$this->_csv_data = null;
			}
			?>
			</div>
			<?php
			fclose($handleread);
		}
	}
 
	/**
	 * Write the output files
	 */
	private function writeData() {
		/* Create the filename */
		$filename = $this->_outputfolder.$this->_original_filename.$this->_filecount.'.csv';
 
		/* Check if the column headers need to be added */
		if ($this->_columnheader && $this->_filecount > 1) $this->_csv_data = $this->_column_headers.$this->_csv_data;
 
		/* Open the file */
		$handlewrite = false;
		if (!$handlewrite = fopen($filename, 'w')) {
			 echo "Cannot open file ($filename)";
			 return false;
		}
 
		/* Write the data to a file */
		if (fwrite($handlewrite, $this->_csv_data) === FALSE) {
			echo "Cannot write to file ($filename)";
			return false;
		}
 
		/* File write is success, now clean up */
		echo "Success, wrote to file ($filename)<br />";
		fclose($handlewrite);
		return true;
	}
}
$csvcut = new CsvCut(true);
?>
</body>
</html>
I need to perform without a script form without clicking on Create files.

Could you anybody help me?
Thank you very much
Vana
Last Edit: 1 year 3 weeks ago by rolandd. Reason: Added code tags
The administrator has disabled public write access.

Re: Automacally dowload and import dats 1 year 3 weeks ago #28048

  • rolandd
  • rolandd's Avatar
  • OFFLINE
  • Fresh Boarder
  • Posts: 14682
  • Thank you received: 613
  • Karma: 296
Hello Vana,

You don't need this CSVI Cutter script at all. You can use this command to download a file
wget -O /path/to/destination/Stock_CSV.zip http://www.my-download-link/
in your cron command. You can import the zip file into CSVI, where CSVI can unpack it.
Kind regards,

RolandD

=========================
If you use CSVI, please post a rating and a review at the Joomla! Extension Directory
The administrator has disabled public write access.

Re: Automacally dowload and import dats 1 year 3 weeks ago #28053

  • vanaveno
  • vanaveno's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 38
  • Karma: 0
1. But I need csvi_cutter because the file is very big. It has about 24 000 rows and I can not to upload all of csv file by csv import.

2. CSV 1.9. can unpack it?
The administrator has disabled public write access.

Re: Automacally dowload and import dats 1 year 3 weeks ago #28061

  • rolandd
  • rolandd's Avatar
  • OFFLINE
  • Fresh Boarder
  • Posts: 14682
  • Thank you received: 613
  • Karma: 296
  1. If you are importing the file via cron it should be OK but it also depends on your hosting. Even if the file cutter cuts it in multiple pieces you will need a bash script to handle the subsequent import of all the cut files
  2. Version 1.9 cannot unpack zip files.
Kind regards,

RolandD

=========================
If you use CSVI, please post a rating and a review at the Joomla! Extension Directory
The administrator has disabled public write access.

Re: Automacally dowload and import dats 1 year 3 weeks ago #28063

  • vanaveno
  • vanaveno's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 38
  • Karma: 0
Hello,
yes, i can to use crontab. But now when i want to use csvi_cutter.php I must click on Create files. I need to modify csvi_cutter.php so that it will without form and without having to click on Create files, but I dont know how. Can you help me what i must change in code?
Thanks
The administrator has disabled public write access.

Re: Automacally dowload and import dats 1 year 3 weeks ago #28069

  • rolandd
  • rolandd's Avatar
  • OFFLINE
  • Fresh Boarder
  • Posts: 14682
  • Thank you received: 613
  • Karma: 296
Hello,

I can't code the script for you, it is more than just a few lines modification. This is really a custom job. You will need to find someone who can code it for you.
Kind regards,

RolandD

=========================
If you use CSVI, please post a rating and a review at the Joomla! Extension Directory
The administrator has disabled public write access.
The following user(s) said Thank You: vanaveno
Moderators: rolandd
Time to create page: 0.700 seconds
Powered by Kunena Forum