I'm trying to write a file to HDFS through Python Script with the below mentioned code.
import hadoopy
import os
hdfs_path = 'data.json'
def read_local_dir(local_path):
for fn in os.listdir(local_path):
path = os.path.join(local_path, fn)
if os.path.isfile(path):
yield path, open(path).read()
def main():
local_path = '.'
hadoopy.writetb(hdfs_path, read_local_dir(local_path))
if __name__ == '__main__':
main()
In this case, to execute the above code I'm trying to pip install hadoopy
package in cmd, but installation fails. Also tried installation by downloading the hadoopy-0.5.0.tar.gz didn't worked. Kindly help me resolving this issue.
Error :
> pip install hadoopy
Collecting hadoopy
Using cached hadoopy-0.5.0.tar.gz
Complete output from command python setup.py egg_info:
running egg_info
creating pip-egg-info\hadoopy.egg-info
writing top-level names to pip-egg-info\hadoopy.egg-info\top_level.txt
writing pip-egg-info\hadoopy.egg-info\PKG-INFO
writing dependency_links to pip-egg-info\hadoopy.egg-info\dependency_links.t
xt
writing manifest file 'pip-egg-info\hadoopy.egg-info\SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<Path>\AppData\Local\Temp\pip-build-4mzcdtzr\hadoopy\setup.
py", line 95, in <module>
ext_modules=ext_modules)
File "c:\python34\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "c:\python34\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "c:\python34\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "c:\python34\lib\site-packages\setuptools\command\egg_info.py", line
180, in run
self.find_sources()
File "c:\python34\lib\site-packages\setuptools\command\egg_info.py", line
207, in find_sources
mm.run()
File "c:\python34\lib\site-packages\setuptools\command\egg_info.py", line
291, in run
self.add_defaults()
File "c:\python34\lib\site-packages\setuptools\command\egg_info.py", line
320, in add_defaults
sdist.add_defaults(self)
File "c:\python34\lib\site-packages\setuptools\command\sdist.py", line 125
, in add_defaults
for _, src_dir, _, filenames in build_py.data_files:
File "c:\python34\lib\site-packages\setuptools\command\build_py.py", line
59, in __getattr__
self.data_files = files = self._get_data_files()
File "c:\python34\lib\site-packages\setuptools\command\build_py.py", line
86, in _get_data_files
file[plen:] for file in self.find_data_files(package, src_dir)
File "c:\python34\lib\site-packages\setuptools\command\build_py.py", line
94, in find_data_files
+ self.package_data.get(package, []))
TypeError: can only concatenate list (not "map") to list
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in <Path>\A
ppData\Local\Temp\pip-build-4mzcdtzr\hadoopy\
Thanks in Advance..!