Skip to content

GitLab

Notebooks
Notebooks

Please review any and all PUBLIC repositories, groups and associate files. These allow anyone on the Internet to access without authentication. Repository and group owners are responsible for their content and permission settings. Go to your project(s), click on Settings > General and expand the "Visibility, project features, permissions" to change this setting.

Commit 83e7ca7e authored by Wigal, Jacob (CIV)'s avatar Wigal, Jacob (CIV) ✨
1 Browse files
Options

removed command-line dependency, added all geopandas drivers support, added... 

removed command-line dependency, added all geopandas drivers support, added input, added helpful readouts for troubleshooting
parent f5800cd9
Hide whitespace changes
Inline Side-by-side
Showing with 65 additions and 385 deletions
geojson_conversion.ipynb
View file @ 83e7ca7e
......@@ -6,73 +6,16 @@
"source": [
"### About\n",
"\n",
"This notebook is for the following functions:\n",
"This notebook is for:\n",
"\n",
" - Converting Shapefiles to GeoJSON files\n",
" - Converting Geopackages to Geo\n",
" JSON files\n",
" - Converting KMLs to GeoJSON files"
"Converting geospatial data files such as Shapefiles, Geopackages, and KMLs to GeoJSON files"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Shapefile/Geopackage/KML to GeoJSON Conversion"
]
},
{
"cell_type": "code",
"execution_count": 155,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Current Working Directory C:\\Users\\Dan\\Anaconda_dan_packages\\cid\\usvi_gis\\usvi-optimzation\\Data\\jeff_good_data_geojson\n"
]
}
],
"source": [
"import os\n",
"import geopandas as gpd\n",
"print(\"Current Working Directory\" , os.getcwd())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Replace /Users/filepath/etc/ with the filepath of the folder your shapefile/Geopackage is located in."
]
},
{
"cell_type": "code",
"execution_count": 156,
"metadata": {},
"outputs": [],
"source": [
"directory = 'C:/Users/Dan/Anaconda_dan_packages/cid/usvi_gis/usvi-optimzation/Data/jeff_good_data_geojson'"
]
},
{
"cell_type": "code",
"execution_count": 157,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Current Working Directory C:\\Users\\Dan\\Anaconda_dan_packages\\cid\\usvi_gis\\usvi-optimzation\\Data\\jeff_good_data_geojson\n"
]
}
],
"source": [
"os.chdir(location)\n",
"!cd $location\n",
"print(\"Current Working Directory\" , os.getcwd())"
"# GeoJSON Conversion"
]
},
{
......@@ -86,343 +29,80 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Running the next cell will make a new folder called geojson within the folder ccurrent working director. Then, it will try to convert any vector layer within the folder ((e.g. if you have three shapefiles in the directory, there will be three converted geojson files). This works for all layers that geopandas can read, which is essentially all normal vector layer types GeoJSON for every shape file. Note: Running this cell will also reproject the data to CRS WGS84 (EPSG 4326) for all files with a different CRS."
"Running the next cell will prompt you to enter the filepath of the folder containing all the files you wish to convert to GeoJSON. Then, it will attempt to convert any geospatial data file within the folder ((e.g. if you have three shapefiles in the directory, there will be three converted geojson files). It will place the converted files in a new folder within your directory called \"geojson\". This works for all layers that geopandas can read, which is essentially all normal vector layer types GeoJSON for every shape file. Note: Running this cell will also reproject the data to CRS WGS84 (EPSG 4326) for all files with a different CRS."
]
},
{
"cell_type": "code",
"execution_count": 159,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Trying to convert arcs_df_ss.gpkg...\n",
"Success: arcs_df_ss.gpkg was converted to arcs_df_ss.geojson\n",
"arcs_df_ss.geojson is now in folder geosjon.\n",
"\n",
"Trying to convert DNodes_10Aug.geojson...\n",
"DNodes_10Aug.geojsonCRS is not WGS84. Re-projecting Data...\n",
"SUCCESS: DNodes_10Aug.geojson was re-projected from epsg:32620 to epsg:4326\n",
"Success: DNodes_10Aug.geojson was converted to DNodes_10Aug.geojson\n",
"DNodes_10Aug.geojson is now in folder geosjon.\n",
"\n",
"Trying to convert DNodes_10Aug.gpkg...\n",
"Success: DNodes_10Aug.gpkg was converted to DNodes_10Aug.geojson\n",
"DNodes_10Aug.geojson is now in folder geosjon.\n",
"\n",
"Trying to convert estatesSTX.gpkg...\n",
"Success: estatesSTX.gpkg was converted to estatesSTX.geojson\n",
"estatesSTX.geojson is now in folder geosjon.\n",
"\n",
"Trying to convert geojson...\n",
"EXCEPTION: geojson is not a vector layer or geopandas readable filetype\n",
"geojson could not be converted.\n",
"\n",
"Trying to convert GNodes_10Aug.gpkg...\n",
"Success: GNodes_10Aug.gpkg was converted to GNodes_10Aug.geojson\n",
"GNodes_10Aug.geojson is now in folder geosjon.\n",
"\n",
"Trying to convert HNodes_10Aug.gpkg...\n",
"Success: HNodes_10Aug.gpkg was converted to HNodes_10Aug.geojson\n",
"HNodes_10Aug.geojson is now in folder geosjon.\n",
"\n",
"Trying to convert ONodes_10Aug.gpkg...\n",
"Success: ONodes_10Aug.gpkg was converted to ONodes_10Aug.geojson\n",
"ONodes_10Aug.geojson is now in folder geosjon.\n",
"\n",
"Trying to convert PNodes_10Aug.gpkg...\n",
"Success: PNodes_10Aug.gpkg was converted to PNodes_10Aug.geojson\n",
"PNodes_10Aug.geojson is now in folder geosjon.\n",
"\n",
"Trying to convert SNodes_10Aug.gpkg...\n",
"Success: SNodes_10Aug.gpkg was converted to SNodes_10Aug.geojson\n",
"SNodes_10Aug.geojson is now in folder geosjon.\n",
"\n",
"Trying to convert TNodes_10Aug.gpkg...\n",
"Success: TNodes_10Aug.gpkg was converted to TNodes_10Aug.geojson\n",
"TNodes_10Aug.geojson is now in folder geosjon.\n",
"\n",
"Done.\n"
]
}
],
"outputs": [],
"source": [
"# Make a geojson folder if it doesn't exist\n",
"current_directory = os.getcwd()\n",
"geojson_directory = current_directory + '/geojson'\n",
"\n",
"if not os.path.exists(current_directory):\n",
" os.mkdir('geojson')\n",
"import os\n",
"import geopandas as gpd\n",
"import re\n",
"\n",
"# iterate over files and try to convert all to geojson\n",
"for file in os.listdir(current_directory):\n",
" print('Trying to convert ' + file + '...')\n",
"def convert2geojson():\n",
" \n",
" # Ask user for filepath of folder containing files to convert\n",
" def enterfilepath():\n",
" while True:\n",
" filepath=(input(\"Enter filepath of the folder containing your files: \"))\n",
" if os.path.exists(filepath):\n",
" break\n",
" print(\"Not a valid filepath.\")\n",
" os.chdir(filepath)\n",
" print(\"Converting all files in\" , os.getcwd(), 'to geojson.')\n",
" enterfilepath()\n",
" \n",
" # Make a geojson folder if it doesn't exist\n",
" current_directory = os.getcwd()\n",
" geojson_directory = current_directory + '/geojson'\n",
"\n",
" try: \n",
" # Load data\n",
" data = gpd.read_file(file)\n",
" if not os.path.exists(geojson_directory):\n",
" os.mkdir('geojson')\n",
" print('Will place all converted files in new \"geojson\" folder.' + '\\n')\n",
" else:\n",
" print('Will place all converted files in existing \"geojson\" folder.' + '\\n')\n",
" \n",
" # Iterate over files and attempt to convert all to geojson\n",
" for file in os.listdir(current_directory):\n",
" \n",
" # Re-project if not WGS84\n",
" if data.crs['init'] != 'epsg:4326':\n",
" print(file + 'CRS is not WGS84. Re-projecting Data...')\n",
" try:\n",
" data['geometry'] = data['geometry'].to_crs(epsg=4326)\n",
" print('SUCCESS: ' + file + ' was re-projected from ' + data.crs['init'] + ' to epsg:4326')\n",
" except:\n",
" print('EXCEPTION: ' + file + ' could not be re-projected ' + data.crs['init'] + ' to epsg:4326')\n",
" print('this error may occur for the following reasons:')\n",
" print('1) there is data that has no geometry that needs to be added')\n",
" print('2) geopandas to_crs() methods breaks for some recent versions of shapley')\n",
" print('RECOMMEND: checking data file or look for implement other method using gdal or ogr')\n",
"\n",
" # Export data as GeoJSON\n",
" filename = 'geojson/' + os.path.splitext(file)[0] + '.geojson'\n",
" data.to_file(filename, driver = 'GeoJSON')\n",
" # Skip folders and listed filetypes\n",
" if os.path.isdir(file):\n",
" continue\n",
" if not file.lower().endswith(('txt','jpg','jpeg','geojson','ds_store','pdf',\\\n",
" 'xlsx','doc','docx','git','qpj','prj','dbf','shx','cpg')):\n",
" print('Converting ' + file + '...')\n",
" \n",
" # Print success message\n",
" print('Success: ' + file + ' was converted to ' + os.path.splitext(file)[0] + '.geojson')\n",
" print(os.path.splitext(file)[0] + '.geojson is now in folder geosjon.')\n",
" print('')\n",
" try: \n",
" # Load data\n",
" file = str(current_directory) + '/' + str(file)\n",
" data = gpd.read_file(file)\n",
" \n",
" # Alert user if CRS is changing with conversion\n",
" if data.crs['init'] != 'epsg:4326':\n",
" match = re.search('epsg:.[^\\']*', str(data.crs)) #use regex to split each dms string into integer components to input in the dms_to_decimal function\n",
" crs = str(match.group())\n",
" print('Original CRS was ' + crs + '...will be converted to epsg:4326')\n",
" \n",
" # Create new filepath with geojson folder and extension\n",
" head, tail = os.path.split(file)\n",
" filename = head + '/geojson/'+ tail\n",
" filename_ext = os.path.splitext(filename)[0] + '.geojson'\n",
" data.to_file(filename_ext, driver = 'GeoJSON')\n",
" #Print success message\n",
" print('Success: ' + file + ' was converted to GeoJSON' + '\\n')\n",
" \n",
" except:\n",
" # Throw exepction for files that geopandas cannot read\n",
" print('EXCEPTION: '+ file + ' is not a vector layer or geopandas readable filetype') \n",
" print(file + ' could not be converted.')\n",
" print('')\n",
"\n",
"print('Done.')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### For Geopackages:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Running the next cell will make a new folder within the folder chosen containing one GeoJSON for every Geopackage file. (e.g. if you have three Geopackage files in the directory, there will be three converted geojson files) Note: Running this cell will also reproject your CRS to WGS 84 EPSG 4326."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"application/json": {
"cell": {
"!": "OSMagics",
"HTML": "Other",
"SVG": "Other",
"bash": "Other",
"capture": "ExecutionMagics",
"cmd": "Other",
"debug": "ExecutionMagics",
"file": "Other",
"html": "DisplayMagics",
"javascript": "DisplayMagics",
"js": "DisplayMagics",
"latex": "DisplayMagics",
"markdown": "DisplayMagics",
"perl": "Other",
"prun": "ExecutionMagics",
"pypy": "Other",
"python": "Other",
"python2": "Other",
"python3": "Other",
"ruby": "Other",
"script": "ScriptMagics",
"sh": "Other",
"svg": "DisplayMagics",
"sx": "OSMagics",
"system": "OSMagics",
"time": "ExecutionMagics",
"timeit": "ExecutionMagics",
"writefile": "OSMagics"
},
"line": {
"alias": "OSMagics",
"alias_magic": "BasicMagics",
"autoawait": "AsyncMagics",
"autocall": "AutoMagics",
"automagic": "AutoMagics",
"autosave": "KernelMagics",
"bookmark": "OSMagics",
"cd": "OSMagics",
"clear": "KernelMagics",
"cls": "KernelMagics",
"colors": "BasicMagics",
"conda": "PackagingMagics",
"config": "ConfigMagics",
"connect_info": "KernelMagics",
"copy": "Other",
"ddir": "Other",
"debug": "ExecutionMagics",
"dhist": "OSMagics",
"dirs": "OSMagics",
"doctest_mode": "BasicMagics",
"echo": "Other",
"ed": "Other",
"edit": "KernelMagics",
"env": "OSMagics",
"gui": "BasicMagics",
"hist": "Other",
"history": "HistoryMagics",
"killbgscripts": "ScriptMagics",
"ldir": "Other",
"less": "KernelMagics",
"load": "CodeMagics",
"load_ext": "ExtensionMagics",
"loadpy": "CodeMagics",
"logoff": "LoggingMagics",
"logon": "LoggingMagics",
"logstart": "LoggingMagics",
"logstate": "LoggingMagics",
"logstop": "LoggingMagics",
"ls": "Other",
"lsmagic": "BasicMagics",
"macro": "ExecutionMagics",
"magic": "BasicMagics",
"matplotlib": "PylabMagics",
"mkdir": "Other",
"more": "KernelMagics",
"notebook": "BasicMagics",
"page": "BasicMagics",
"pastebin": "CodeMagics",
"pdb": "ExecutionMagics",
"pdef": "NamespaceMagics",
"pdoc": "NamespaceMagics",
"pfile": "NamespaceMagics",
"pinfo": "NamespaceMagics",
"pinfo2": "NamespaceMagics",
"pip": "PackagingMagics",
"popd": "OSMagics",
"pprint": "BasicMagics",
"precision": "BasicMagics",
"prun": "ExecutionMagics",
"psearch": "NamespaceMagics",
"psource": "NamespaceMagics",
"pushd": "OSMagics",
"pwd": "OSMagics",
"pycat": "OSMagics",
"pylab": "PylabMagics",
"qtconsole": "KernelMagics",
"quickref": "BasicMagics",
"recall": "HistoryMagics",
"rehashx": "OSMagics",
"reload_ext": "ExtensionMagics",
"ren": "Other",
"rep": "Other",
"rerun": "HistoryMagics",
"reset": "NamespaceMagics",
"reset_selective": "NamespaceMagics",
"rmdir": "Other",
"run": "ExecutionMagics",
"save": "CodeMagics",
"sc": "OSMagics",
"set_env": "OSMagics",
"store": "StoreMagics",
"sx": "OSMagics",
"system": "OSMagics",
"tb": "ExecutionMagics",
"time": "ExecutionMagics",
"timeit": "ExecutionMagics",
"unalias": "OSMagics",
"unload_ext": "ExtensionMagics",
"who": "NamespaceMagics",
"who_ls": "NamespaceMagics",
"whos": "NamespaceMagics",
"xdel": "NamespaceMagics",
"xmode": "BasicMagics"
}
},
"text/plain": [
"Available line magics:\n",
"%alias %alias_magic %autoawait %autocall %automagic %autosave %bookmark %cd %clear %cls %colors %conda %config %connect_info %copy %ddir %debug %dhist %dirs %doctest_mode %echo %ed %edit %env %gui %hist %history %killbgscripts %ldir %less %load %load_ext %loadpy %logoff %logon %logstart %logstate %logstop %ls %lsmagic %macro %magic %matplotlib %mkdir %more %notebook %page %pastebin %pdb %pdef %pdoc %pfile %pinfo %pinfo2 %pip %popd %pprint %precision %prun %psearch %psource %pushd %pwd %pycat %pylab %qtconsole %quickref %recall %rehashx %reload_ext %ren %rep %rerun %reset %reset_selective %rmdir %run %save %sc %set_env %store %sx %system %tb %time %timeit %unalias %unload_ext %who %who_ls %whos %xdel %xmode\n",
"\n",
"Available cell magics:\n",
"%%! %%HTML %%SVG %%bash %%capture %%cmd %%debug %%file %%html %%javascript %%js %%latex %%markdown %%perl %%prun %%pypy %%python %%python2 %%python3 %%ruby %%script %%sh %%svg %%sx %%system %%time %%timeit %%writefile\n",
"\n",
"Automagic is ON, % prefix IS NOT needed for line magics."
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%lsmagic"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Microsoft Windows [Version 10.0.18362.720]\r\n",
"(c) 2019 Microsoft Corporation. All rights reserved.\r\n",
"\r\n",
"(pyomo_test) C:\\Users\\Dan\\Anaconda_dan_packages\\cid\\usvi_gis\\usvi-optimzation\\Data\\jeff_good_data_geojson>mkdir geojson\n",
"\r\n",
"(pyomo_test) C:\\Users\\Dan\\Anaconda_dan_packages\\cid\\usvi_gis\\usvi-optimzation\\Data\\jeff_good_data_geojson>for filename in ./*.gpkg do ogr2ogr -f \"GeoJSON\" \"./geojson/$filename.geojson\" \"$filename\";done\n",
"\r\n",
"(pyomo_test) C:\\Users\\Dan\\Anaconda_dan_packages\\cid\\usvi_gis\\usvi-optimzation\\Data\\jeff_good_data_geojson>\n",
"(pyomo_test) C:\\Users\\Dan\\Anaconda_dan_packages\\cid\\usvi_gis\\usvi-optimzation\\Data\\jeff_good_data_geojson> \n",
"(pyomo_test) C:\\Users\\Dan\\Anaconda_dan_packages\\cid\\usvi_gis\\usvi-optimzation\\Data\\jeff_good_data_geojson>"
]
}
],
"source": [
"%%cmd\n",
"mkdir geojson\n",
"for filename in ./*.gpkg do ogr2ogr -f \"GeoJSON\" \"./geojson/$filename.geojson\" \"$filename\";done\n",
" except:\n",
" # Alert user if file conversion failed\n",
" print('EXCEPTION: '+ file + ' is not a geopandas readable filetype') \n",
" print(file + ' could not be converted.' + '\\n')\n",
"\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### For KML:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Running the next cell will make a new folder within the folder chosen containing one GeoJSON for every KML file. (e.g. if you have three Geopackage files in the directory, there will be three converted geojson files) Note: Running this cell will also reproject your CRS to WGS 84 EPSG 4326. You may recieve a warning such as \"The output driver does not natively support DateTime\"."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!for filename in ./*.kml; do mkdir -p geojson; ogr2ogr -f \"GeoJSON\" -t_srs crs:84 \"./geojson/$filename.geojson\" \"$filename\";done"
" print('Done.')\n",
" \n",
"convert2geojson()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
......@@ -441,7 +121,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.10"
"version": "3.7.3"
}
},
"nbformat": 4,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment