Export Tables from Access to PostgreSQL

Table of Contents

Export Tables from Access to PostgreSQL

Hello,

I wrote a program for a client in VBA using the MS Access database to help him calculate personnel salaries in 2014. Now, I am re-creating this “Personnel Management System” that is already working for the same client, with Python Django and PostgreSQL.

While rebuilding the project with Django, I had to import the tables from the Access database into PostgreSQL. When I searched for programs on the Internet, I found third-party applications. I couldn’t trust them either.

After all, I’m a programmer and I have a powerful language like Python.

Why shouldn’t I create the codes that will make my job easier?

While writing these codes, I transferred personnel information from Access DB to PostgreSQL. In order to ensure the confidentiality of personal data, this article includes the data with staff salaries.

As a result, I can add all the records to the table related to PGAdmin with the insert script. When I type all the records under the query window and run the query, all the records will be added. Provided the data is compatible with PostgreSQL of course. What I’m going to do here is to process the data in Access DB in Python and print the query scripts to a bit text file.

Following these steps will make it very easy for me to export Access tables to PostgreSQL while doing a small scale project.

  1. Saving the relevant table from Access DB as an Excel table,
  2. Updating column names in Excel table ( if the column names defined in PostgreSQL are different from those in Access DB.)
  3. Editing the data (Date fields must be in PostgreSQL date format, NULL values, etc.)
  4. Saving the Excel file as a CSV file.
  5. Converting the saved CSV file to a data frame with the Python Pandas library, looping the row and column values of the data frame, writing the data as an insert script and saving the script as a line in a txt file,
  6. The scripts from the txt file and pasting them into the PGAdmin query window and running them .

Saving the relevant table as an Excel table from Access DB:

After selecting the relevant table, we select the “External Data” menu. Here we click on Excel. We save the Excel file by saying OK.

Updating column names in Excel table

Bunu yapmak zorundayız, çünkü PostgreSQL’deki ve Excel dosyasındaki sütun adları tutarlı olmak zorunda!

Organizing data

Only the “ change_date ” column in the Excel table is in date format. We have to save this format as text. The PostgreSQL format is “YYYY-MM-DD”. We can use Excel’s translate-to-text function for this . The formula will be as follows. Of course, the excel I use is in Turkish. English users should write “YYYY-MM-DD”!

YYYY – > Year, 2026

MM-> Moon, 03

DD->Day,23

We convert the entire column to text with the fill handle. Then we paste it again in the “ change_date ” column. The Excel file will change as follows. Salaries are also in Excel currency format. We need to convert that into numbers as well.

Saving the Excel file as a CSV file.

Excel dosyasını UTF-8 formatında csv dosyası olarak kaydediyoruz

Converting the saved CSV file to a data frame with the Python Pandas library, looping the row and column values of the data frame, writing the data as an insert script and saving the script as a line in a txt file,

Let’s create a new project in Anaconda Spyder . The appearance of the “100_salarychange.csv” file that we have saved in this project folder will be as follows. Here, decimals in numeric values are separated by “ , “ (comma). In this python, “ string ” will appear as a value. To avoid this, use the text editor to replace “,” with “.” The image is below:

The codes below will convert the csv file to SQL script and save it to the file named sql.txt.

				
					# -*- coding: utf-8 -*-
"""
Created on Wed Jan 25 21:01:06 2023

@author: Murat Uğur KİRAZ
"""

import pandas as pd
import numpy as np
import os

#Parameters
access_to_postgre_csv_file = "100_salarychange.csv" # write down csv file
postgre_table_name = "public.personnel_salarychange"

# delete file if out output data exists
if os.path.exists("sql.txt"):
  os.remove("sql.txt")

# Import data from csv to pandas dataframe and if there is null data inside, fill them with "Null" string
data = pd.read_csv(access_to_postgre_csv_file, sep=";")
data = data.fillna("NULL")

#get the column and row number of the data
column_number = len(data.columns)
row_number = len(data)

# we will have column names and add column_str variable
column_names = list(data.columns.values)
column_str = ""
for column_name in column_names:
    column_str += column_name + ","
column_str = column_str[0:len(column_str)-1] # this will delete the last comma , ;)

# loop in columns and rows, append each row in appropriate PostgreSQL insert script format
# save in the sql.txt file
for i in range(row_number):
    row_str = ""
    for j in range(column_number):        
        #row_str += str(data.iloc[i,j]) + ","
        #print(type(data.iloc[i,j]), data.iloc[i,j])
        if type(data.iloc[i,j]) == np.str:
            if (data.iloc[i,j] == "NULL") or data.iloc[i,j] == "1900-01-00":
                row_str += "NULL, "
            else:
                row_str += "'" + str(data.iloc[i,j]) + "'" + ","
        else:
            row_str += str(data.iloc[i,j]) + ","
    row_str = row_str[0:len(row_str)-1]    
    sentence = "INSERT INTO {}({}) VALUES ({});\n".format(postgre_table_name, column_str, row_str)
    file = open("sql.txt" , "a", encoding=("UTF-8"))
    file.write(sentence)
    file.close()
    



				
			

The appearance of the sql.txt file will be as follows. The result we want😊

Copy the scripts from the txt file and paste them into the PGAdmin query window and run them.

Just copy and paste 😊Hooray python.

Result: Success!!!

Blog summary

This article describes how to export a table from Access DB to postgresql via excel and python.

About the Author

Other Posts

My Thesis
Murat Ugur KIRAZ

Conclusion

In this blog post, the Flooding Attack, Decreased Rank Attack and Version Number Increase Attack in the RPL protocol were trained and detected by “Decision Tree”, “Logistic Regression”, “Random Forest”, “Naive Bayes”, “K Nearest Neighbor” and “Artificial Neural Networks” algorithms.

The test results for the attacks were compared, as a result of the comparison, the Artificial Neural Networks algorithm with an accuracy rate of 97.2% in the detection of Flooding Attacks, the K Nearest Neighbor algorithm with an accuracy rate of 81% in the detection of Version Number Increase Attacks, and the Artificial Neural Networks with an accuracy rate of 58% in the detection of Decreased Rank attacks algorithm has been found to show success.

Read More »
My Thesis
Murat Ugur KIRAZ

Interpretation of Machine Learning Values

I continue to share how I did my master’s thesis titled Comparison of Machine Learning Algorithms for the Detection of Vulnerability of RPL-Based IoT Devices, my experiences in this process, and the codes in this thesis in a series of articles on my blog.

So far, I have provided detailed information about the RPL protocol and the attacks that take place in the RPL protocol. Then, I experimented with Flooding Attacks, Version Number Increased Attack, and Decreased Rank Attack, extracting the raw data and making sense of that raw data. I compared the results of experiments with weak knots with statistical methods.

In this section, I will interpret the numerical results of the attacks we detect with machine learning algorithms.

Read More »

Share this post

LinkedIn
Twitter